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ABSTRACT 



A -new diagnostic modeling^ system for automatically synthesizing a deep ; 
structure model of a student 'sT misconyeptioos or bugs in his basic mathematic % 
skills provides a mechanism for explaining why a 'student is making a mistake 
as opposed to simpl^-^lrd^tif ying the mistake. -This report consists of three 
sections: Th^^^rst provides examples of the problems that must be handled 
by a diagnoa/^c model, M: then introduces procedural networks y as a general 
framework for representrag tTie knowledge underlying a skill. The challenge 
in designinjg this reOTesentation is to find one that fat^ilitates the* discovery 
of misconceptionav orJb existing in a particular student'^ encoding of 
this knowledg^ jf^ie &e^midse£tion discusses some of the pedagogical issufes 
that have emerged from the useots. diagnostic models within an instrucjtional , 
system. ' Thisydiscusslon is f ramedV in the context ok a computer-based 
tutoring/gaming system developed ti teach students and student teachers how 
to diagnose bugs strategicall^as/well as how to provide a better understanding 
of the underlying structure 'of /^ithmetic skills. The third section describes 
our. uses of an* .executable network as a toolyfor automatically diagnosing student 
behavior, for automatically genWa^ng "jii^nostic" test^, and fbr judging the 
^diagnostic quality of a given ex^iiK Incited in this section is discussion 
of the success of this system in di^guosing 1300 school students from a data 
base of 20,000 test items. 
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In our ICAI Report No. 4 entitled "Aspects of a Theory for 
Automated Student Modelling" we discussed the use of a procedural, 
network for modelling both t*he correct and inc'brrect encodings of a 
given procedural skill. We also alluded to how this modelling 
scheme could be used to create ^j,agno3tic ^ode-la of a student's 
misconceptions r> but di4 not describe' how these models could be 
inferred from a student's behavior. This report providers a, 
descriptifon of the procesq and decision rules us^jfli to ^utopa^t ical ly 
infer (or construct) a diagnostic models of £ student from his 
Observed correct and incorr^^ answers on a given test. Thi3 report 
also describes t^he result of an experiment that used this diagnostic 
madelj.ing system to automatically construct a diagnostic, procedural . 
model for each of 1 300 students from their answers on an arithlnetic 



screening exam. - ^ . 

< The first two sections are expanded* v^i^sioris of what appeared 
in chapter one. of the ICAI Report'. No ; 4^^, ' They are .included her e in 
order to provide a framework for understanding the more technical 
details of the diagnostic mo'delling' process as described in tho^ 
-latter part bf this report. . « / 
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DIAGNOSTIC MODfiLS FOR PROCEDUR^AL BUGS IN . BASIC MATHEMATICAL SKILLS 



"If 

as 




you can both listen to children and accept t^heir answers not 
things ta-- just be // jiTdged right or wrong but as pieces 'of • 
.information which may revegil ^what the child ^s. thinking- you will 
have taken giant st e p/// toward becpming a master teacher rather 

\ than merely a ' disjseminatOif of information," J /A • Easley, Jr, & 

Russell Zwoyer ; 

;rntroduction 

"One oS the 'great esV^talelnts o'f. teachers is • their ability to 
synthesize ah accurate | " p'ict ur e'^ , . or ^ model, of . a student's 
misconc6pt ions from^ th^ meager evidence inherent in hlsverror^, * ^ 
d.^tailed model of a studen^'/s knowledge, including his 'misconceptions , 
is a prerequisite to successful j^emediation,' The structure, use, and 
inference of such models fotj procedural skills in mathem^-tics is * the 
topic of „ this , papar.^ Ijh , part ici^lar we shall describe some initial 
■ efforts in the development //and use of a representational ' technique 
called "' "procedural networks" \ as the * framework for constructing 
d 1 a g n o 3 1 i c Xk o dels - - .i , e • ,/y models^ that capture a student's common t 
misconceptlions or faults behaviQ.r as simple changes to (or mistakes 
' in) ?a correct model of tWe underlying knowledge base. By b,eing able 
to synthk^^e such deep/i structure diagnostic models - aot-Qjiaticallv , we 
"move a step clbser to be'^ihg^able to provide both' a teacher and -an 
instructional system wm(i3^o^t only supplying an identification of v£hiL^ 
mistakes a, S(budentl /s making, but also an/ explanation of yhy those 
m^istakes a^e being jna^e. ^Such a system a^s.o* yias profound implications^ 
foF^XBstik^, since a/student *«ieed no lounger' be evaluated solely hn the> 
num,ber of e^ors appearing on his test,\but rather on the 
misconcept ions » whio/n he harbors. 



fundamental 



This paper .//con si,st of four sections. The first provides 
examples of the pnoblems that must be handled by a diagnostic |model,.' 
It then. .■introdu||es procedural^ networks as a^general fr^lewox^k for 
representing the 
to design a 
misconceptions 



V. 



this knowledge , 



i ss^ues 
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that ' havi 



[nS§^wl\edge underlying a skill, ^'^he; challenge herie 
I ^presentation that f acilitaT:es^ the " discov^r( 
Ibugs e'xist^ng in a particular . student^ s en^od: 

secon^ sectdon dis^cusses some of the 'P^da/gogic-al 
emerged from ^ur use of diagnostic models within a-n 
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Instructional s^^tem., This system is at ^^computer-ba^e^d tutoring gajne 
develyDped to^ teach both . students and^^ student teachers about .the 
str'ategic diagnosis of bugsT, The third section describes our uses of; 
procedural network as\ a tool r f.or automatically diagnosing student 
behavior^ for automatically g^neratTTng ' "diagnostic"- tests, and for 
judging/ vthe diagnostic .quirity of a* given exam. Included in this 
sectl-on| is a di scussian ;of the 3uccess' of thrs system in diagnosing 

1300, gijade- school student's from a data bas6 ^of 20,000 test items. The- 

' '* 'I ■ ■ '■ . ■ ' ^ ' • " 

last section discusses some future resea^f'ch directions. • 

: ' . ' ' r, " • ^ " ' ^ ■/ , . ' ■ ' \ - 
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^' Section I 



nTAGNQSTIC MOnRLf^^OF BASIC SKILLS 



■ \ 



•VThe issues^^->.^dressedMn this paper -arose/^ from an, invent igat ion 
of t\e ^procedural ^kills necessary to perfbij^ high-^ j^^^^^ algebra.* 
These Iskills include" not only the^ generally r>cpgWized rules of 
.aigebr4'^: -but al so .such normally implicit . skii^ls as .>i^^^^^^^ of 
formula^, the parsing of expressions and the determination of -rwhich 
rules to apply next [(Brown, and Burton, \ 1975;/ ;3r0Wn,; Collins and 
Harris-, N -1977; Mgtz; .19.78). For this paper: however ,. we^llmlt our, 
discussion -to examples encompaasing arithmetic skills so tha^ . we can 
conc^elTt^^ critical ideas of diagnosis without /the n.^e for a 

large number of algebraic rules. " LlAiVing our ^examples to arit'hmetic 
also provides a- compelling demonstration of the difficulty^ of the 
diagnostic task; 1 1 -"c 1 e a r 1-y^ a^mlsm a more^ diff icult i,t 

is to ' diagnose what Is w^rpng wi't^^ stiidejit'^s' method of per^ a 
task to form a diagnostic modeP than; to perfarm the task 
itSelf . m •••particular',', it^ Is; ;no;-. gre^a>t . challenge t6 add ar subtract 
^two numbers but , as we shall see^ misconceptions in th^se. 

^s^ame skills can be quite subtle* . - / ' \ 

Let. us Vconsider. di-agnos]in'g what ' is w-rong Wi'th^ the arithmeti(^' 
skills ('procedures ) pf^ l'^a c^^ ©f stutients . - We s^^aiNl start with V 

case study in which ye' eftcami'ne J^ive "snap '^L^.t^^^ y student ' s\ 

perform/rtc seen ob/a. homework 

Before Jproceeding , look at ^•the_^_^^ m^P shots ^nd 'try to 

. discover the student ' s bug- * ^ 



Sample^oTS^^ stu4,eQt ' s wprk: , ... ' J ^ ' ■ 

^.41 ' 328^^^ 989, ^ ■ ^. '^'^ 

|o :T33*&^ \ -105^ -229. , : . 



Once you have .discovered the " bug\ try testing your hypptjiesis by 
"simulating" t^e -buggy student so as to predict.y his results on t.he 
folloVinfe two'^ test problems; . ■ . ' 



-•V 
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Th^ bug is really qai.te simple . In, computer terms , the stude'nt , 
after determining^ the' carry ,.Jftigr gets to reset the "carry register" to 
z.ero and hertoe the amount carried is' accumulated across the ^columns. 
For example, '*in the stii'dient • s second pr obi em , ( e . g . 328 + 9^17 equals- 
.i13'45) he proceeds as follows : . 8+^ = 1 5 so he writes 5 and carries 1 J' - 
2 + 1=3' plus the-one carry^is*^^, lastly 3 + 9=12 but that .one carry from; > 
the first column is still there it hasn' t been"^ reset ^* so v addin'g ' 
it in to. this 'column giv6s 13. If; this is the bug, t hen ^ the 'answer s 
to the test problems will be 1361 and 700. This bug is. no^-^O} absurd- 
:when o.ne considers tHat a child might use his finger s^o rem'em ber the 
carry and forget to bend his^ ^f^^ngers, or counters after each* 

carry is added. * . \' - ^\ 

. . .A- cbmi?i|on assuitfptiori ampng 'teac^her s /is that students do-' not follow 
pr.o:cedures /very *welland that grr'atiCN behav ior is the primary ' cau:^fe of; 
a- , student's inability to. perform- each step correctly. Our exper ie'nce . 
: as been thaENstudents are r*emarka^ly competent procedure .'followers, 
but _ that t/hey oft'en^ follow t)iQ wrong DroGeduygts , One - case ^ 
encountered *lyast year is oT special interest . - The . stuAen)/ proceeded 
thrquglrry^ year with. his t^^her thinking' 

thjat^ he ,w^s 1 eol>h4^bl t'i.ng^ in iiis arithmetic performance,^ 

frs" , far as the /t eacber. . wa^ • con'cerned there was no systematic 
explanation for hi s error s. Here is a sample of his work:- 

-I r 9 • * 8 ^6. 8-. j 9 17 19 ' 87 ' \ 

■ , T^;- if - If.. Tf- A. <,# ^ .. 

36^ 67? ?^3/. 27,493 A" ; 



36^ 67? ?S3, - 27,493 , '-797 



•There is a clue/ to the nature of his bug in the number of ones 

in his answers. Every J:ime^' the ad(iition of a^ column involves a carry,. 

a one my St eriousf^. appears in that column; he is simply writing down 

the - carry digit anfi f orgett ing about the. upits digit! One might be 

misled" by, 17+8 which ndrmally Involves . a carry yet is added 

correctly; It would s4jem that he is able'to do simple additions by a 

completejj^y different procedure^, possibly by cpuryting frOm the* 

larger number o,on his fingers. . 

• \ ; ^-^ • ■ " ■ {. ■ ■ 

The\ manifestation <or this stutien^t '-s simple bug carries over / to 
^ , . . ■ '•■ ' ■ ■ - ^ ■■ . ■' ■ . .■ • ^ 

other types of " probl'ems, which involve addition as a subskill. What 

answer would -he give for the forib.i^ing? 



A ramily hs(s traveled 2975 mile^ on a tour of the U,S, They h^ve 
1828 miles, to go. How many mile^ will they hav^e traveled at the'^ 
end of their tour^ ^ - 

He Vcorrectl,y , solved the, word problem to obtain the, addition problem 
^975 -t- 1875 ,--Jb.Q which he answered^ 3191.^ Since his work w^s done on a 

scratch sheet .^ -.ferk;e==^# teacfher only saw the answer which is, of .course, 

wrong. Aj3 a result, the teacher assumed, that he had trouble with word 
prxDblem^. as well as with arithmetic. 

When we studied this same student's work 'i,n other arithmetic 
procedjires, we discovered a recurrence of the. same bug. Here is a 
saB^p±9^f his work in multiplication: 



68 73M '5-43 758 2764 

,27 31 ^ 



x46 xrr X2Q6 x2?6 

"792 141 144 ,.2731 



Several bjigj3-;are manifested here, the most severe one. being that his^ 
multiplication algorithm miiflics the column behavior of his addition 
..---alrgoT^i t hm ,- But notice that fhe bug in, his addition algorithm is also 
'present \ in his multiplication procedure,' The 

;"determine-uni t-and-carry " subprocedure bug s'hows up in both his. 
multiplication and addition. For' example, to do 68jc>^ in- the first 

.column he performs 8x6, gets 48 ar^ the'n' writes down-' the "carry", in 
t'his . case —4, ignoring the units digit,* Then , ..vh^, multiplies 6x4 to 
get 2 for the second column. All along he has a comple'te ^nd 
consist ent procedure fo;" doing arithmetic. His answers throughout all 
of '.his arithmetic; work are far^from random; in fact, they display 
near perfection with respect to h j,3 way of doing it, 

A First^ Approxitfat i on to Representing Procedural Skills 

For a computer system to be capable of diagnosing a^berrant 
behavior such as' Jthe above, the procedural sicill being^ taught must be 
represented in a form amenable to modeling incorrect as well as 
correct subprocedures of that skill,. Furthermore, the representation 
must '^allow the intermixing of both the correct and incorrect 
subskills, - so that the modei can capture tlios^e parts of a-skill that 
are, correct as well as those that are wrong,- The ^ breakdown of the 
skill into : shared subskills can also account for the r ecurr ence * of. 
- similar errors in different skills. We introduce the' term diagnostic 
model to mean a representation of a stud'ent's procedural' knowledge or 
skill that' depicts his internalization of a skill as a variant of a 
correctversionofthatskill. 




In addition to satisfactory representat ibnal._^ techniques, th'e 
diagnostic modeling task requires that the repreij^ntat ion of a 
particular correct ^skill ''make explicit much of the tacit knowl-edge 
underlyijig the skill. In particular, the correct model must contain 

. all of jihe knowledge that can possibly tre misunderstood by the student 
or else some student misconceptions "will be beyond the* diagnostic 
modeling capabilities of the system. For example, if the model of 
^addition doesn't include the transcription of the problem," the system 
would never be able to Jiagnose ar student whose bug is to write' 9's 

. that he later misreads as 7J s . " 

y The technique we use to represent, diagnostic models ia^ a 

^ ^r^cedural network . (1) A. procedural network consists of a collect r 
of procedures with annotations i/n'which th^ calling relationships 
between procedures are made explicit by appropriate links. Each 
pro'cedure node has two main^ parts: a conceptual part representing t.he 
intent of ^t he^proced ure , and' an aper at ibnal • part consisting of methods 
for carrying out that intent. The methods ' (also - called 
implementations), are programs that define how the Results of other 
procedures are combined . to satisfy the intent ot^ a particular 
procedure .( 2 ) Any procedure can have more. than one 'implementation , 
thus providing a way to model different methods for . performing the 
same skill. ^ F,or most skills, the ne t worjc^ r e pr e sen t at ion takes the 
form of a lattice. Figure 1 presents a partial . breakdown of.^a portion 
of the addition process into a procedural network. Conceptual 
procedure nodes are. enclosed i'n ellipses. The €op .procedure in the 

( 1 ) Thi s t erm has been used by Earl Sacerdot i ( 1 975 ) to describe an 
interesting modeling technique for a partially ordered sequence oT 
annotated steps in a problem solving "plan" as well as" for specifying 
control information. Our use of procedural nets coincides with his on 
this latter feature but differs from and is l^ess developed than . his 
with regards to "plans". . , 

(2) The language we have used to define these programs is LISP. The 
par/tl^GsUl ar programming language is unimportant from a theoretical 
st^ndposint because an implementation is" hon- in t r ospec t abl e . . The 
model ingvaspects of the network must occur at the conceptual procedure 
level. FVr— example , the implementa^tion of the- subt ract ion facts table^ 
look-up procedure in the computer is necessarily different from that; 
in the student. However, the conceptual properties of the procedure 
are the same in both. Those aspects which are the same, the invoking 
of other procedures, the values returned ,^ the relevant side effects, 
are included in the network, while, the implementation details that may 
differ are "swept under the rug" into the ' program . This is not a 
limitation, as any " im pi emen t at ional issue" can be elevated to the 
conceptual level by creating a new conceptual procedure in between the 
existing ones. Tne distinction between conceptual and implementation 
details also allows skills to be modelled efficiently at different' 
levels . 
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lattice is addition, (3) Two of the possible algorithms for doing 
addition are pr^ented as alternative methods under" the conceptual 
iiod^ for addition. In method 1 , the standard algorithm, the columns 
are added from right to left with any carries ,being written above and 
included in the column sum of the next column to the left. In method 
2,.t^he columns are . added from left to right with any carries being 
written below the answer in the next column to the left. If there are 
any c^arries, they must be added in a second addition. Notice , that 
the'se 1?wo methods share the common procedures f or cal.culat ihg a column 
sum and wrlting./a digit . In the answer bvjt . differ in the .procedure, 
t^ie.y ' u^se when ' carrying' is 'necessary , . One stnuctjjrt^al aspect -of the 
networlG'is to nja'ke -e.xplici t any subproc edur es that, can be potentially 
shared by*^ several -hif^her level procedures,. - . ' V / 

^ The decomposition of a complex Skill in to all-, of ^ i t s'- cqnc ep t ual 
'p\p<5eduir es terminates in some set of. primi^tives that r^fleat . assumed 
ei^'ents ..of an - underlying computational ability, " For addition, 
typical assumed primlti-ves are recognizing a digit , writ ing a' digit, 
and knowing the concepts of right, left, etc. The complete procedure 
network explicitly ' specif ies all the subprocedures of a skill and -can* 
be evaluated or "executed", the/eby^ simulating tl^e skill for any 
,glven set of^inputs, -By itself, t*his network merely provides .a 
computational machine that performs the skill and i's not of particular 
in^port. However, the possible, misconceptions' in this skill are 
represented in the network by incorrect im^plementat ions Associated 
with procedures in the decompoaitf^.n called "bugs" ,( 4 )^ Each buggy 
version contains incorrect actions taken in place of the correct ones. 
-An extension to the network evaluator enables the switching in of a 
buggy version of a procedure ' that allows the network to simulate the 
behavior of that buggy subskill, .This feature provides a 
computational method for determining the external behavior of the 
underlying bugs, • • 
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{\\ This sii^pllfied representation demonstrates only those^ ^ f eatares 
of the procedural network particularly relevant to the diagnostic 
task. The actual breakdown into subprocedures may be different in a 
particular network and will be considerably more detailed, ^^r^r^o 
f M ) The term "bug", is borrowed fr om 9omput er. science where it reiers 
to a mistake in a computer program. 



Inferring a Diagnostic Model-of the Student . 

•\ ' •• • • ' ■ ^ .r ' ■ 

The 'problem of ^la^nosing U deep structure failure In a student's 
knovrledge of a procedural, skill can now be accomplished, 'at least 
theoretically, in a -straightforward manner. Suppose, as in the 
examples on page 5, are prcvided with several surface 

manifestations of a deep structure misconception, or bug, in the 
student's addition procedtire. To unco^v^r . which possible subproc^dures 
are at fault, we use the network to simulate -the 'behavior of buggy 
subprpceduries t>ver tft% se^| of problems and note th'ose whic^ generate 
th(? same . b^^h-avior as exhibited- by the s^tru-den^. To catch ^ student's 
ia'irfconce{)t ions that involve more than one faulty subprocedure , we< must ^ 
be ^able to simulate various combinations of. bugs,' A student may haV^S. 
a,. bug in his^ carrying^; pro(yedure as well, as believing that 8+7 is IX. (a / 
bug in his addition facts table). To m^pdel hji^ behavior, both buggy 
versions ' must' be s^used^ together , ' „ deep- structure mo^del of the 
stxident's errors is a set of buggy sub proc ed ure s that, . whenV invoked > 
replicates those errors. Each buggy version has associated 
informat ion . such as what the underlying causes^ of the bu'^j^^a^ have 
been, as well as specif ic fremediations , explanations , - interactions , 
and exampless^of the bug all of which may be^ used by a tutoring 
system to help correct ' the student ' s problem, (.5) 

Many' technical questions are raised by. the above^ brief overview . 
of how to "infer" a diagnostic model. We have deferred a more^ 
detai,l.e<d discussion of these questions until Section 3 /In favor of a 
more gener.al disQussiop of pedagogical, ramifi cat ions of ; the procedural 
network mo^del of procedural skills. We begin this discussion with a 
description of the procedural, network for a simple » skill'' 
subtraction , • 

A^lfocedu ral ^Network for Subtraction ^ ^ 

As an example of the surprising amount ' of procedural knowledge 
neededl - to ' perf or'm a simple sk'ill, let us consider a m^re complete 

(5) West (1971) has broken down the diagnostic teaching task into .four ^ 
gteps: ' ^?(i) distinguish between, conceptual arfa 'careless errors; (ii) 
identify" the exact nature of the conceptual errbr (bug); (iii) 
determine the conceptual basis (cause,) of the bug: and (iv) perform 
the appropriate remediation.. The system we describe has been directed 
towards ^problems (i) .and (ii). The buggy implementation nodes in the 
network provide the proper places to att'ach information, relevant to 
problems^ (iii) and (iv), / . • 
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network . representation of the subtraclion of two fiumj)er s . ( 6 > Figure 2 

shows the links of the procedural network ^or subtraction that 

indicate 'which subprocedUres a procedure may use. • 

* • 

The topmost - nod'e represenjuS^ th^ subtracti^ *of two n-digit 
numbers. It may use the procedures for .settling up the problfem, 
transforming it if the tiott^om number is greater^ than^'^the ' top, and 



sequenQjji.g through each column performing the column subt-R^^t ion * 'The 
imp^mentat ion of the .column subtraction procedure ' has Jo ^"count for 
cases where borrowin'^g is necessar-y and may call upon many other 
subprocedures,, including' taking the borrow. from t^ie correct placed 
scratching 0 and writing- 9 if that place contains^a zero, aji<^ so on. 
An important subprocedure is the facts taible look-up whi^^b^ allows- any 
of^ the simple .^^^arvDthmet ic facts: to be; wrong. The ' facts^ table 
subprocedure Jirs" called during the addition of 10 to a column, digit , 
during th.^^subtr ac t ion of 1 from a / cdlumn digit in a borrowing 
operation, as well as during tbe £(ubt rac t ion of the' bottom from, -the 
top 'digit in a column . \i 

^In principle, each^of these lisubprooeduFes couTd^ have many buggy 
v-ersions associated with. it. (7) An example of a- common bug is to 
calculate - the column difference by subtr^ctin^g the smaller digit from 
the larger regardless of which is on top. In another bug, the set-up 
'[Procedure . left- justifies ' the top and' bottom numbers so that when the 
student is told to siibtradt 13 from 18-5, he gets 55. An interesting 
aspect of^^fTie Left jus t i f ic at i on bug is that when the student is faced 
with s.e'emingly .impossible problems ( 185-75 ) he may be inclined to 
change the direction in, which ' he subtr-^ borrowing 'from right to 

left instead of from le^Tt to right, orvxo change his col^imn difference 
procedure to larger minus smaller, thereby ^eliminating the need to 
borrow.. Thus, there can'e^xist 'relationships be twee. n-. .bug s^^:;, such that 
one bug suggests others. ^ A major challenge- in' identifying the 
procedural breakdown or description of a skill is to ^a^e the network 
handle such ramifications and interactions of. multiple bugs. 

I To provide a feeling for the >;range' of "answers" that* can come 
from simple underlying bugs, we have included ..in figure 3 the 

(6) We have chosen just one of the several subtraictipn algorithms (the 
so-called "standard" algorithm), but the ideas /presented here apply 
equally to others and can handle mul t i pi e me t hods ,as well. 
(t) In our current subtraction network, many subprbcedures have only, 
one bdggy version, but some have as many as fifteen. The av^erage is 
three or rour. . * ■ ' 
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Figure 2 

^PriQ^dural Network for Subtraction 



1 



slibt 



'answers" to a suDcraction problem (15300-9522) using some of the bugs- 

'* . *^ ■ . • . . . ■ 

in the procedural network for subtraction. For exaifrple,- the answer 

-14222 was generated by 'the bug that subt.ractdj ; the smaller digit in 

each column from^ the larger. Appendix H gives brief "explanations' of 

bugs that" would^ 'generate each of the answers in Figure. 3. 



■ ; - ■ 

Figure 3 
Manif esta.tions of Some" Subtra 



ion' Bugs 



Eac^ pf these ' answers is 'the result of^ using a subtraction 
with a single hug. ' 
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As / can be seen in App.endix 4, a particular answer to a given problem 
can have more .than one explanation since several distinct bugs can 
generate the same answer. A special case is that a student may harbor 
many misconceptions; and stili get the correct answer to a particular 
problem! ' 



The Power of Simulating .Bugs in the Network 

Given a 'procedui^l network like the one :j.n Figure 2, it' is not, 
{ . ' . ■ . 

always obvious how bugs in any particular subproc^edure or set of 

subprocedur es will be manifa«te.d on the surface, that is, in the 

answer. • Some of the- complicating factors are that a'^ingle buggy 

subprocedure can . be used by several higher-order procedures in 

computiM5 an answerer that two bugs can have interactions with each 

other. Alhese'' factors -are further complicated by th^ fact that not all 
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sample problems will manifest ^Xl of- the ppssible^ sfimptom^. If asked 

to make predictions abou]: the sym^ptoms of' a given bug, people often 

determine the symptoms *by conside-ring onl'y the skills or subprocedur^s 

used.^ifa solving one particular sample ^* problem . As a result; t^iy 

often miss symptoms generated / by. other procedures' tHiat "can, in 

principle, use^-the given buggy ^bprocedure but which, becausejof the 

. • ■ ^ ' * ' #^ 

,characteri^t ies of tli'e Articular problem, were^n? t . called or/; iff a 

diff erent f sample problem^ad been chosen^ ^it ^^might have baused the 
particular fairtt-y-^--?trbp^ocedare , to have been used for a different 
purpose^^ thereby gen^ating different symptoms* ^ .These obsery.at ic^d, 

^irst led* us t^o coni^id.er the^;allie * o^r^'simulat icw . to sy at e'ma"t;i,^ally 

\ verify a conjectured bag, \ u ^ ^ \ ' • 



Section 2 
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BUGGY -An Ins^truc tional Game fpis Trj^iniug Student Teachers and Others 

•V . .. V . . 

As'^we .saw in the first^ section^ it is often diFficult to Infer a 
student's bug from his- answers. It was the .need for teachers to 
appreciate, this problem and to develop strafegie^ /f.orr^^pping. with the 
p'<)-ssible range of student bi^gs that led us. to coristruct a game called 
BUGGf.- '^UGGY is. ^a comp^Jterl zed game based or^ the- d$||gnostic 
fnt'feract ions of^a tator arid^a student. The computer plays the part of 
an err-ant student whose malady must be identified by the diagnostician, 
( a ^ role either: played by an T'individual or txy a /team).- \ Th 
diagnost^ici^Ctt^iS; pr^sen'ted with ah Arithmetic homework^ probl emi, ,that 
the ^"student:?* has done . inc^r r ec 1 1 y*. . The " st uden t ' s" beljav. i^ 

^eqera'ted' using the pVoceci'ural neCVprk and shows the symptom's of an 
underlying bug in 1" one of th'^ arithmetic ^ subprocedures . The 
diagno^st ician » s job is to discover exactl:^ ^what is the underlying, 
misconoep^t ion V He does this by providing strategic test problems, for 

(th^e ''stydent" to solve. The problems given by the diagnostician are 

^answered by -the . computer apting as the student usioSft^e bugged 
procedure. ' ' . > , * . 

The computer also acts , as arbiter in the evaluation of the 
diagnostician's theory. "When the diagnostician indicates that he has 
discovered the bvi«, he is -asked to describe it. Then, to make certain 
that he really hap found th^ bug/ he is given a five-problem test 
during' which he must answerHhe protilems in the same errant way that 
the student wo.uld. In other words, \he must simulate the* "student's" 
bug.' When the diagnostician's concept of th^ bug is suf f i c i-ent 1 y well 
formulated that he is able to predict 'the buggy answer on all five 
problems, he is considered to y;^ave diagnosed, the student and he 
pro'ceeds to a new bug. ' i_ . . ' 

We ^have found it beneficial to allo*w teachers, to teaja up while 
p^laying the diagnostician's role in BUGGY. The. interaction between 
team taembers encourages verbalisation of the decision-making processes 

x'involved/in the game. ' ) 

The following •is a protocol of a team of ^ teacher s using BUGGY. 

The dialogue is interspersed * .with commentary a'bou^t.j the ^important 



issuea-. Those lines typed by the team are' marked' with a ver t ical^line. 
in the' left margin. < ' - - 

• r ■ , ■ / 



Protocol of a Team Using BIIOGY ^ 

HERE IS AN EXAMPLE, OF THE BUG 



\ 



WELCOMfi TO BUGGX^. 
,1 HAVE CHOSEN ft BUG 

17 • - * 



V 

J 



The ^system presents an example in which the stirdent' doesn't 
get the riff ht.. answer . ' At present these initial examples are 
-stored in the network linJced to the i)uggy' implementation . 

NOW. YOU CAN GIVE *ME PROBLEMS TO DETERMINE THE^BUG. ^ - _ 
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b'V the 
similar 



The -team generates problems that are answered 
"student". N-otice that the team's test problems are 
tfo th6 initial exam'ple 'geiierat e'd b'y BUGGY. This is usuall 
a po.or ' diagnost ic strategy as it may' leave other symptoms o 
a bug undiscovered. " » 



Gat the 



bug ! 




V 



The team f^rms an initial hypotrhesis and tells the system 
thev have discovered t-he bug. The system now asks the team 
to ^describe the bug. and then tests them on their 
understanding. Asking for a description forces the team to 
verbalize their sometimes fuzzy idea of what the bug is and 
often makes them realize how ill-defined or vague their idea 
actually is. In the present /system/Mhe descriptions are 
saved but no attempt is made by the system to understand 
t hem . ^ 7 

.■'■/./ ^ ^ ' / 



PLEASE DESCRIBE THE BUG 



Student 
worked" 



adds 
in. 



the numl^er to be carried 



This cryptic description 1/s 
oJ>served.' Describing bugs/ in 
dif .f ic^*!^ . . . ' What the team/ is 
column a'dds to m'ore than 9, 
digi t ' are added ' together 
answer-; For example, In th 
prob-lem; when 7' and 5t ar 
units, digit) are added to K/et 



last .column 



rpical 
Eni 



of 'those ' we have 
glish is surprisingly 
trying to say is that if a 
he unit s digit and the ^carry 
and -the result is written as the 
first column of the example 
added, 1 (the carry) and 2 (the 
The same uwith 8+6, 2+9 



are added to gjet 3. The same cwiti. w-ru, ^t:;, 
and 3 + 9-. (This bug could eas^ily arise if tshe student were 
' using, method 2 from figure/1 and mistakenly^ wrotJe the carry 
under the "'same column.) ' 

The hypothesis does acdounjb for the observed behavior 
Unfortunately, it is not /'the' student's bug. Notice the 
* possibly humorous, probably disastrous results that wo,uld 
have transpired if the te/am had ^topped ' at this point /arid 
tried to tutor tl^e st/udent on the basis of their Initial 
. , ; belief! 

HERE ARE SOME PR0BL?M'^ TO TEST YOu/r THEORY ABOUT THe'^'SuG 
WHAT IS.v^ I ■ ( 

2 1 



To ensure that the team has found the correct 'bug, the 
- system produces sever/al problems ' for the team to solve. At 
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present, BUGGY has .a heuristic problem 'genei>^tion jirogram 
.^hat gives a broad grange of "different" problems. TFor 
;i€xamples which the bug is very localized'*, such as 

•requiring a 0 in the top number, a special problem generator 
c^n linked to. the bug and will 'be used insteard.. ) These 

pr*b*iems kre then filtered to try to' get three* prc^blems 
-which cause sym'ptoms and two tha't>>^give correct answers. 
This simple- routine does surprisingly' well /ak, producing 
critical ^ >problems. $o well, --in fact,' that a more 
"intelligent"' routine which determined, the consistent set of 
bugs and then filtered the t^st problems to split the set 
was discarded as unnecessarily t imS-consuming . Hiowever, 
this mare -intelligent technique^ — bsised on techniques 
discAissed- inssegtion 3^ — might be reverted to' as our theory 
of^proO'edural bugs becomes progressively mo^re subtle. 

The team*'s hypothesis is revlaJLed/^s incorrect by the fifst 
of, the five test problems. . ByO'G^K presen^t ijts atisVTer ,ja?id 
then allows the '^eam to gnter mTSre problems. ^/ ' 
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THAT'S NOT THE BUG "-I'MA VE . 
^SING MY BUG : . ' 

f / * 

■ 21 • • . /■ ' ^ 



TRY -GIVING^JIE SOME MORE EXAMPLE^. ' 



/ 




51 



jLio:^^ 




Aften^ failing the fc>3t , ''th^^eam gives a giieater range 
problemg, specif icallir tl^osfe «|girig .bltfger ■ numbejrs. .How 
it.. . that .'51 + 17a7 • can add up.,to*- 2 1 .? . j,By "_hj;oa'denT»« the sc 



Of 

is 
scope 



ar estample^^^ pnqbl^ma .they. *arl able', to^ reevalu^tfe. - their 
initial hypofeh,esis,. dismiss ,it '. 'and. -develop a^bth^^r ope; that 
is complete and . coivsl s t^nt .»wi t-h res pec t tb the new. ein-oFS^ 



s ^Got the bug! 



PLEASE DESCRIBE THE -BUG YOUj. THINK I HAVE. ' . 

H , ^ .... J 

The student is\ adding all the, digits together 



HERE ARE SOME PROBLEMS TO- TEST YOUR THEORY ABOUT THE' BUG 
WHAT IS: 



/ 



I 



• 33 



1091 

+.60 




^ 28 
+ 70 



■II 

-IT 



) 



This. time the team 
given, the descri 
network. 



eam's hy^pothesis passes the test and they ai>fe 
iptionr or the bug stored in the 'procedural 



VERY* GOOD. MY DESCRIPTION OF THE BUG IS: 

THE STUDENT ALWAYSS^^SUMSVUP ALL THE DIGITS WITH . NO REGARD TO' COLUMNS, 



A variant of this game assigns a cost to each problem the us'^er y 
poses in thei process/ of trying to discover the 'bug. At the*^ same ' time, 
a much higher, cost' is assigned for. clainjing to have "disjcovered" the ~ 
bug but then failing to -paSs criterion on the computer-generated test 
pr^oblems. A correctly structi%re'd coi^t algorithm encourages the u;3er 
to tHiink- carefully abo'ut the diagnostic ' value of, each^-pr^blem he 
creates ami ;not tos^jump to a conclusion .too quickly or too ^slowly . , . 

: ■ ■ > ■ ^ • ■ ^ ^ . ^ ■ . . . ^ 

FgCl^fiQPilg^l Iggy ^? fbr^ T^^P . her , ^ • \ 

One application of BUGGY and the . "diagnostl^c models" view\ of 
procedural skills lies in the domain of teacher^ trai-ning. The 




realization that errors that appear "random" are oft 



manifestations of / a systematic underlying bugj^^is a m'ajor conceptual 



the surface 
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breakthrough for many stud^nl: teachers 
■•- v' ^ ' I - • - 'l6" -'' 



All too of teilVs^ beJravior that 



X 



appears to "^be random hap a simple, ^intelligent., ati.d complete, 

.underlying explaneefeion By proper diagnosis, .remediation for a* 

' student can directed to>/ards*' his. specific weaknesses. (TJxe 

Jl4mportanc.e of simply admitting that/t.h-fere may. exist underlying bugs 

can not -be overstressed . Witjjout appreciation of this fact, a teacher 

must view failure ^on a particular prob^lem as either carelessness or 

total algorit^hm .-failure/. . ,In the fl^rst ' . case, .^the ' pi^edicated 

remediation, i-s giving more probl.eimS,''- 'while in the^second, ±i is'^going 

' ^ ■ ■' • ' ' ' ^ , ' ■ * • , ' ' ^ *. • . ^ - ' 

-o^er the .entire algprit'hm . ( 8 ) ..v. When 'a student ' s bug (whic)i may^onPy 

manifest itself occasionally) is not recognized- b-y the t estcher , ' the 

teacher ^eXp\ains the errant behavior as carelessness, lazinesfs, or 

■ w-or;3e thereby 'of ten^*. m.istakenly lo werVing , h j^s opinions of thfe student Vs 

capabllliTie? . V".' ' ; . ' / ^ .:, ' ^ ' * . u • 

From the student ' s "Vieuwpoint the situation is ^ much worse .^^ He is; 
following^ whdt . he believes to be ' thfe- cbVrect algorithm and , seemingly 
at raitdpm, .gets ^ffarked wrong ^ This situation can be exacerb'a.t.ed by 
, improper ddag.nosi^* -Fc>r » example , Joh^^nlfe subt r act s- 284 from 437 and*' 
' gets 253 as an an-swer. "Of course", says the teatJher "you forgot to . 

subtract- M ^ from. 4 in ,the hun'dred.s pflace when you borrowed." . 
' Unfortunately Jiohnn.ie ' s .algorithm is to subtract the smaller digit in 
each column ^f rom th^ larger . Johnnie^ do6 sn ' t have any idea what the 
teacher is. talking about (he a.ev er . rbc^f rowed/' !) -arid feels that he myst 
be very stupid indeed not to understanTd. The teacher agrees with this 
assessment a^ none of his r.emediat jLon has had aqy effect on. Jo^hnnie^ s 
performance . > * . « 

BUGGY,* in its present , form, ^reaents teachers with exampl es- ' otf 
buggy • behavior and provides practice in diagnosing' . the underlying 
causes of, errors. Using BUGGY, telchers gain experience in forming 
theories about the relat ionship "betj/een the symptoms of a bug and the 
underlyl^n^^ ^b^g l^"self.\ This ^^/perience can^tie qultiyated to make 
teachers a\are that there are methoxjs or strategies that they can use 
. to ciiagnp^ "bugs properly. \ 

fact, there i^are a irffmbTr of qt>rategv bug^ that teachers may 
have in forming ' hypo^eses about a student 's .iiisconceptions . , That *is, 
the task of diagnosing .a student- is a ,pro(>>dural skill and as such ^i,s 

■ ' * , •■ * * ' - ■■ ' ' . 

TEl In computer programming met aphors, ithlS approach corresponds to 
the debu&ging act ivities of resubmitting the program' because , the 
computer, must have made a 'mistake and of th'rowing the wtiole program 
away and starting over from sjcjratcjh and writing a new program. - 
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'.susceptible to mis^Learning by the teachers, A common strategy bug is 
l} to jump too quickly to one hypothesis . Prematurely focusing on one 
; hypothesis can . cause a teacher to be unaware that there may b§ other 
,'competling hypotheses that are possibly more likely. A common 
psychological effect of this approach is that the. teacher generates 
problem's for the student that confirm his hypothesis! In some cases, 
iteachers may believe their hypotheses so' strongly th^t they will 
ignore contrary evidence or decide that i>\ is. merely rando^ noise. 
One general diagnostic strategy that aVflNids this pitfall is-the 
, technique of differential diagnosis (Rubin, 1975 ) in which one always 
generates at lieast two hypotheses and, then chooses test problems that 
separate them. ^ - * ... 

Another common strategy bug 4s to lock OTito only* one type ■ of 
symptom. For example, one student ^teacher was gi.ven the initial 
example (A), after which he proceeded to generate example problems (B) 
and (C): . ' " ^ 





B , 


■c 


19 


33 
+ 6 


81 


1?9 







At this point, he concluded that the bug was "writes the bottom ^(dtigit 

^ after the top number." But his hypothesi-s faile4 when he wa'S given 

the first test pr.oblem: ' 

' 8 ' • ' / 

to. -which hfe- responded 8 1 2 The bug is that single Higit operands are 
linked on to the end of the other operand, so that the .correct buggy 
answer is 128. By presenting examples only with a shorter bottom 
digit, 'he i had obtained what seemed to be confiririlng evidence of his 
hy pothe si s • 1 ' A general rule which could be employed to avoid this 
' fixation is that whenever an example of incorrect behavior has an 
' asymmeirry (length of top and bottom n,umbers), then try an example' with 
the apymmetry reversed. , Ui-ing this rule, the teacher would alsp 
generate problems with ^larger top numbers before \ h-e reached a 
conclusion. BUGGY provides an environment in which teachers can 
experience the ramifications of not employing rules and strategies 
.during d iagnpsis . ( 9 ) 

(9) On<e j®b for an "intelligent" tutor for BUGGY is to recognize anJ 

- 1 8, - 4^ '.^ - • ^ • ^ 
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. ^ * Another important issue, concern-s the relationship between "the 
.language used to describe a student's errors. and its effect on what a 
teacher should do to remediate it, [is the language able to convey to 
the student what he is doing wrong? Should we expect teachers to be 
able to'" use language as the tool for correcting, the buggy^^'aTg^i thms 
of students? Or should We expect teachers only to be able to 
understand what the bug is and attempt remediation with the student 
with things like manipulative math'tools? The following descriptions 
of hypotheses given bjf student teachers, taken from protocols of 
BUGGY', give a good idea of how difficult 'it is to express procedural 
ideas in*- English, The descriptions in parentheses are BUGGY'S 
prestored explanations of the bugs, 

"Random errors in carryover," (Carries only when the next column in 
the top number is blank,) 

"If there are less digits on the top than -on the bottom she adds 
columns diagonally," (When the top number has fewer digits' than the 
bottom number, the numbers are le f t->j ust i f i ed and then added,) 

"Does not like zero in the bottom," (Zero subtracted from any number 
Is zero , ) / 

"Child adds first two numbers, correctly then when you need to carry in 
'the s6cpnd set of digits, child addsXnumbers carried to bottom row then 
adds third set of digits diagona>lly finally c?irrying over extra 
digits." (The carry is written in the"" top number to »the left of the 
column being carried from and is mistake^^^ for another digit in the top 
number , ) ' . * 

I' ' ■ ' , . 

"Sum and carry all columns correctly until get to la,st column. Then 
takes furthest left digit in both columns and adds with digit of . last 
carried amount. This is in the sum," (When there are an unequal 
number of digits in the two numbers, the columns that have a blank are 
'filled with the left-most digit of that number.) ■ . ^ 

Even when one knows what the bug is in terms of being able to 
mimic it, how one going to explain it to"" the student having 

problems? Considering the above examples, it' is clear that anyone 
asked' to solve a set of problems using these explanations would, no 
-doubt, ^ave real trouble. One* can imagine . *a stude^it^'s frustration 
when the teacher of f ers an^explanat ion of why he is getting problems 
mar ked- wrong , and the explanation is as. confused and unclear a>s these 
are • . ■ _ ' ' • . • \ ■' , 

For -that matter, when, the correct procedure is described for the 
first time, could it, too, be coming across so unclearly! ^ 

The problem of adequately describing^bugs is further complicated** 
by Another surprising, fact: Fundamentally different bugs can cause 

- ( . ■ 

point out places where instances , of general rules and strategies 
should be used.. This possibility is discussed in Section M, 
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Identical behaviorl In other words, there can be several' dlstirict 

ways of Incorreqtly performing a skill ^tha.t always genera^te the same i % 

"answers". For example, here is a set of problems: 

18 186 298 89 

-^64 



"W iili' "?fef 



T 



One possible bug which accounts for these results is: the 
columns are added without carries and the left-most digit in the 
answer is the total number of 'carries required in the problemx^ .In 
this case, the student views the carries as tallies to be counted and. 
added to the left of the answer. But another equally pl^si,ble bug 
also exists; the student places the carry to the, left of the next 
digit in the top number; then when adding that column instead of 
adding the carry to the digit, he mistakes it as a tens column of the 
top digit --so that when adding 298 and 169, in thfe second column he 
adds 19 to 6 instead of 10 to 6 . This generates the same symptoms. 
So even when the teacher is able to describe clearly what he ' believes 
is the underlying bug, he may be addressing the wrong- one. The 
studeiit may act.ually hay^ either one of these bugs. (10) IJ' > 

An, Expgriiqent ,M.gAng BUGGY Student T^ach^rs 

* - / 

To determine BUGGY' s Impact on student teachers, we had a group 
play, the game des'cribed in' the beginning of this section. The goal of 
the experiment was to explore whether exposure to BUGG.Y significantly 
improves the student teacher's ability to detect regular patterns af 
errors in simple arithmetic problems .( 1 1 ) The subjects wer.e 
undergraduate education majors from Lesley College in Cambridge. 
Their exposure to BUGGY lasted approximately one and a half hours, 
during which time both addition and subtraction bugs were presented,. 
The effects of their exposure to BUGGY were nieasured by comparing each 
subject's performance on the pre- and post- exposure debugging test, an 
example of which is given in Appendix 1. The detailed analysis and' 
discussion of the experiment is beyond the scope of this paper but has 
beeji described in a technical report (Brown et al,'1977 ). Briefly, 
though, ''the results of the experiment showed- that exposure to BUGGY 

( 10 ) This possibility leads t'o ah interesting question concerning how 
one can "prove" that two^ different bugs entail /logically the same 
surface manlfesta tSLjO n s . ^ 

(11) We woulcj like'^o thank ,Dr . Mark Spikell of Lesley College for his 
assistance in tnis qndeavor. 
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significantly improved their aUS^lty to detect regular patterns of 
errors . 

, We al so ' invest igated the qualitative issue of: what the student 

teachers felt they gained from their .exposure to BUGGY. To assess 

their impressions, we convened the,, entire group .after th^y had 

finished using BUGGY. At that gathering, we first asked them to 

•■ ■ ■ n r ■ , i ■ 

write their responses to two questions (discussed below), and then we 

taped a final group discussion in which we sought their reactions to 

using 'BUGGY and their suggestions for its deployment with school-aged 

students. The following week, their professor, whjj) also participated 

in the initial ex^per iment , held a second group discussion and reported- 

to US the Qonsensus, which wa^ consistent with what they had written. 

Appendix 2 lists all the written response^ to the question "What 
do you think yoU learned from this experience?". All 24 responded 
that they came away with something valuable. Many stated that they 
now Jippreciated the "complex and logical thought process" , ^hat 
children ofteh use when doing an arithmetic problem Incorrectly. "It 
makes me aware of problems, that children have and they sometimes think 
logically, not carelessly as sometimes ti^achers think they do." "I 
ne\;^r realized the many diffArent ways a child could devise his own 
system:to do a problem." They alWo sl;ated that they learne^cl bet.ter 
procedures for diatjovering the Underlying bug. "T learned that it is 
necessary to try many dif f erent^/types of' examples to be sjare that a 
child r^eally underatands. Different types of -difffcultles arise with 
different^ problems ." A^nd. finally ^^VThe types of. analyses necessary^ to 
'"^ debug'^ st^udent er^robs^. pnw, th^ aq^d pencil) seem- more 

*i^f Ib'ult^^ than.^ wi'th' tW' :Cpm;pute> ) <B^at -that doesn' t make any; sense . 
Yfre* arial^^^^ ougiifc'^ td" b^^^^^^ sarrie . Perhaps' the computer motivated- 

my anai:yt loal .^alitiva'i^^Kyg ■ . ■ . " " ' " . 

Appendix. 3 ^li^,<s.^^^^ to th.e question "What 

is your Veactld^r.tb,%to^^^^ "BUGGY could bemused to 

sharpen a teacher • s awareness of diff erent • difficulties j^'lth addition 
and subtraction." They felt th^t it might be of use in grade schodl , 
high . school, or' Vith special needs, students., or even as a "great 
experience in beginning to play with computers." • 



Pedagogical Issues More Specific to Middle School Stucfents 

We feel that all of the Issues discussed' above are as important 
for sohool-level students^ as they are for teachers, ''ihere is great 
value in introducing young students to procedural notions. The BUGGY 
system provides a;/ well controlled envirop^^(ent for such an 
iht roduct ion , as well as one that can be easily , integrated into a 
standar d curricu],^ . Vi^'v AT so note - that for a cjAddle school student to 
play the diagnos t ic i an » s'rol e require s his studying- the procedural 
skill per se (i.e., its structure)* as opposed to merely performing it., 
This experience can be especially important as' students begin^gebra, 
which is their first exposure ^o, procedure "schem^r!^ By prese^ing 
procedures as objects , of study, BUGGY can thus be used to explore the 
powerful ideas of hypothesis formation, debugging, debugging 
strategies,^ and* so on. Of course, such a use requires more than just 
the BUGGY game . , 

Another reason for having students develop a language for. talking 
about procedures, processes , ^ bugs , and so forth is that such a 
language enables ^hem to talk .about , -and -^think about , the underlying 
causes of their own errors. This facility is important in its pwn 
.right, but it also gives a student the motivation and the apparatus 
for stepping back and critiquing his own t^hinkdng, as well- as saying 
something interesting and. useful about his errors. TJie difficulty \n 
getting students to test the plausibility of their own answer (such as 
by estimating it) may be due to the ^lack of any appropriate 
introspective skills that the.student could ^use pnce he knew his 
answer was wrong. ' • ; , . - ^ • 

An impprtant^ ancillary , no"n-mathemat ic.al benefit of a student's 
involvement with BUGGY is exposure to the Ictea of Vole 'Hevisrsal . ( 1 2 ) 
To communicate effectively with others, children must learn not only, 
language, but also the use of "social speechV speech that takes* 

in'to account . the' knowledge and p'erspecti ve of another, person (Krauss 
and Glucksberg , ^ 1.977 ) . Piaget uses the. term* "childhood egocentrism" 
to describe the child ' s inability to detach hiinself from his own point 
of view and to take into consideration another's perspective. 
jAlthough Krauss and Glucksberg agree that egocentrism plays"" "a large 
part .ffi very young , ch i Idr en Vs speech, they believe that even in older 
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(12) This idea is at tf ibutablef to ■ Tim . Barclay , " at tEe Cambridge 
■ Friends School, who has been 'experimenting with various uses of Buggy 
- with sixth through eighth graders. ' 

: oi « 
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children the ability to -role-play breaks down when t hey ' are, faced with 
a diemanding cognitive task. We beliey,e that taking on the v^ewpoint^ 
of the errant student by analyzing another 'is mistakes can provide 
valuable practice in role-playing in a demanding situation and can be, 
beneficial to the development of "social speech" . 

" ^ome Results on Us ing BUGGY with 7th and Sth^^Graders 

' To explore the effect of the BUGGY game on T'^th and 9th graders 
and to discover what type of additional instructional material and 
ac'tivities must back up the use of jthis comput er- basi^d system', we 
plaped a terminal running BUGGY- in a . classroom. The tea^bher provided 
a short introduction to the game and the notion of bugs and then the 
;.3tudents'( 1 3) were free to use the system during'^ the term. During this 
experience, we noticed , the f ol lo wing^henomena . When the students 
^"T^Tzt started trying to discover the underlying bugs in ^BUGGY , their 
• most comrnon reaction was that upon seeing the mistakes of the 
simulated buggy student was to exclaim "Wow, how dumb and stupid this 
kid must be." However', after a week or so of exposure to BUGGY, the 
students' reactions changed from |beliefving that the simulated student 
was dumb' to' one of appreciating HhatN^here was, in fact, a systematic 
explanation for'what this student was doing. They began *to see that 
he^' .^iit) had fundamental misconceptions as opposed to being just 
stupid. Th'i,s result is particularly exciting, since it '^paves the way 
for students to see their own faulty behavior not/ as being a sign of 

•their stupiditA but^- as a soGlnce of data .from which thpy can 

• • • ^ • t ■ // • • • - '■ 

understand t4ieir\0wa/^er ror s . ' 

Unf pr tuna'tedy ^ . we have as- ^ yet > no , "data concerning the 

transferability "of ' this awar e'nesia , " an^ whether or not 'it will lead to 

^students' being more capable , and .willing to look over their own work 

, for errors . ' ' \ - . , 



(13) All ot the* students participating in this activity had already 
mastered the procedural skills being "bugged". In fact, we have 
severe res^ervat ions about the advisability of younger students using 
this particulBT game .es.pedially if, they haven't mastered the? correct 
versions of th^e skills comprising t^he network. > "owf^^T ' ? 
extensions^to BUGGY (discussed in the last section) that make it 
appropriate for school children in the process of learning the given 
procedural skills. . - 



^ Section 3 



^PIQMATEI^ PIAGNOSTIC MODELING SYST-fiM ^ . ' . - 

^^^^In this section", we describe a diagnbstic system that is based on 
/our arithmetic procedural network, recently completed by Burton, The 
hotion of modeling procedural skills as pyecUtable networks and then 
, expressing all potential^ misconceptions as buggy versions of 
subprocedures in ^e network provides a technique f or ef f ic i ent ly 
determining the consequences or symptoms of a given collection of bugs 
on a set of problems^ It therefore has the potential of diagnosing • 
and explaining all of* the procedurally incorrect answers foj* any ffiv^n 
problem. ,.For example, as. we incficated in Section 1, giy-en -a 
procedural network for addition and an addiction problem ( like 35 + 782), 
all the bugg^ subpr ocedures as well as combinations of buggy 
subprocedur es , 'can be inserted and then executed in the addition^ 
procedure qne by one sa that all the possible buggy answers t^b the ' 
problem are\^ generated). Those answetrs can then{^be compared ^to a 
student's work to de^jt^rmine po^ssible explangit ions of^"^^~tiie student ^s 
particular misconceptions. In this way, pne iqight use the procedural 
network to diagnose a student's basic misconceptions, over an eptire' 
homework assignment or an arithmetic test._ 

Ttrere ane, however, ^sfever'^l complications ► wi th this simple 
paradigm for diagnosing a student. One ip that the student who has 
developed a novel singleton bug (as op*po3ed to one that arose out of^a 
combination of "primitive" bugs) will not be diagnosed. Another" is * 
that students' do *make "random" mistakes (presumably as' many while* 
following an incorrect ^ procedure #s, a.' correc^t on^T^^jiliat could 
erroneously lea.d, to the'' exclusion of, his bug or .the . inclusion of 
anothfer bug th^at happened to coincide with his "randomness" . Finally, 
blindly^ considering all possible combinations of bugs can lead to a 
combinatorial -explosion of possibilities. In what follows, we shall 
discuss some solutions to these pr'oblems. . * ^ 

^ , 

A Deep.-Structure" Data A nalysis Tool ' 



3*As a first step , in developing techniques foj automatically 
diagnosing a student's errors., we sought a large data base of gtudent 
answers on some arithmetic test- We started put this way for two 
reas^ons: First, .an analysis of the student errors not already 
explained by the existing network suggests** extensions' to the network 
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th^Lt .may have been overlooked. Obviously, if the BUGGY system is- 
going to^ be useful arff a diagnostic tool, its arsenal of bugs must be 
very extensive! Second, onc^i a "complete" network of bugs h|as been 
constructed, 'analyzing a large data base provides some evidence of how 

- many student errors are procedural rather than "careless" errors. 
Such an analysis also indicates how consist enVly students apply buggy 
procedures. S)o they use the buggy progedure every time it is 
appropriate? W^fien they 'get a correct answer does that contradict the 
use of their b'u£gy procedure, or is It just that the buggy procedure 
for this problem ' ^does producerj^'thQ correct answer? Thi^ analysis - also 
reyeals which procedural errors occur most of^n, and ^ji what 
combinations. Answers to these questions not only influence the 

' design, of the diagnostic system (as we will see later) but also have 
an impact on how that system could be used. , 

. We forturfeBt^ to be able to obtain a large collection of 

pertinen^t data(i4) already in computer readable form. This data 
stemmed'wfrom an achievement test administered in Nicaragua to 4th, 
•5th, ahd'6th graders. Tt^ere were HO different test versions, each 
consisting of 30 pro*blems combining both simple-a^d complex addition 
and subtraction problems. One version of the test is^given in Figure 
4. The. makeup of each test fpllowed , a complex profcedur'e discussed by 



Friend (1976) 



' Figure 4 
A. Sample Test 




7 
■9 



9 
+ 6 



99 
-79 



257 
• 161 



43 
-41 



353- * 
-342 




213 
21 
"+40 



156 
873 
■311 



-6^3 
-221 



521 

50 2- 
B!40 



8 
54 
+ 9 



81 

'-.17 



-I 



31! 
53( 



505 
743 
12 
+ 35 



77 ■ -705 
18 -9 
+ 47 



)17 10038 
i39 . -4319 

+ 5 . . . 



864 

\ 

+ 3 



10050 
-98 



700+ 
-94 



(14) T'he data used in this study were made avallabl-e by the Institute 
for Mathematical Studies in the Social Sciences at Stanford 
University. The da.€a~ were collected in Nicaragua as part of the 
Nicaragua Radio Mathematics Project supported by Contract 
A10/CM^ta-C-73-40 from U.S. Agency of International Development.^ ^We 
are grateful tQi Barbara Searle for allowing us access to the data. 
See Sear,ae ef al •( 1976 ) fo^ a description of the, NRM project. 
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Admittedly, these data . have the-? limitation that the particular 

results derived from' the|m ^are notx ^nec9ssarily generalizable to 

AKnerican schools.. Althou^gh the procedures taug^ri^for addition and 
subtract ion--^re similar, the environmental and cu^ural ^^experiences 
of ^ t he ^ ^y 3 1 u d e n t s are quite different . Nevertheless, trhl^ data base 
pr.oviiHes a convenient starting point for this research as well as a 
generUl idea ^ of the percenta^e^ ' of s.tudents making procedural as 
opposeiKto "random" errors. In additipn, the methods we devised to 
analyze ] these data wouid apply equally w^ll to data collected Under 
^jOther cincum.st ances . ; ' . . 

^ TheFrncni ess of -Organizing the Data ^. 

The data base available- for this study was large 19>500 
problems performed by 13(?0 students. We linuited ourselve^ to 

ucorisiderat ion of 'just the si^ibt raction"* problems on the test be_cause the 
addition problems "ihcluded some in- which three or more numbers were to 
be add^d, a condition that can produce • error s not modeled in the 
present addition network.^ >. ' \ ; « ' 

As a first step/ we extended the BUGGY system to produce "bug 
comparison" tables for^ students as shown in Table 1'. These tables 
summarize how well the^ student^.'s behavior can be explained' o;^ 

, predicted by. a simple bug. in the network. The format of a table l"^ as 
follows: The problems with the* correct answers appear at the top of^ 
each table. ^ ' The student's answers appear on the next line using the.^ 
convention 'that "-" indicates a correct student answer. Each of the-** 
remaining lines provides the nam^ of a bug 'and the answers produced by* 
the 'assumption that the student had this and onl y ' th;Ls- bug . ( 1 5 ) • For 
each of these lines a means that the bug predicted the student's . 

incorrect answer.- A "»" means that the bug in that row would give the 
correct ans'wer and also that the student got the correct answer.^- 
Thus "•" and "»»»" indicate places of agreement between tlje student's 
behavior and the' simulated, behavior of the bug. An "!" m'eans that 
the bug would give the correct answer, but that the student gave an- 
incorrect pnie.(l6) A number which appears in a bug row is the/ answer 

( 15 ) Descriptions of the bugs appear in Appendix 4. \ , • 

,(16) In both the "•" and the " ! ^ cases, the ) bug has not manifested., 
itself in the , answer 'as an error. For , exaiyple , If a buf is 0-N = 0, it. 
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Table, 1 
An Initial Biig Comparison 



'able 



8 
3 



' 99 353 .633 
■79 3H2 221 



81 4769 257 6523 "103 731 5 1/039 705 10038 10060 7001 
17 0 161 1280 64 653^6 / 44. ' 



4319 



9 8 94 



5 20 1 t 412 " 64 4769 ' 96 5243 
Student answers: 
- -* - - ' 98 - 4^1 8 

*FaRgET/BORROW/OVER/BLANKS: ^ 

»STOPS/BORROW/AT/'ZERO: 

' • * * • . • ,1 • ■ I . • 



39 
1^9 

139 

49 



•DIFF/0-N=N : 
• « « « 



I' 



77^ ' 995 .696 , 57 19 9962 6907 

7/58.1095 706 14319 10078 7097 

««« «*«« 15719^ 10062 7007 

««« ««« 671^9 10062 7017 

"I I 99 7 8 I 



•ADD/INSTEADOF/SUB; 
•1 1- 1 78 695. 85.4 



««« 



7803 1 67 1 3851 1 083 Tl'4 1 435J' 1 0 1 58 7095 



that the bug would give, when it is different from both the student'ja 

answer -affd the correct answer. (17) . ■ 

■ Initially, any ^ bug timt explained any of the student's behavior 
. ■ , L.--' ■ • 

(i.e., generated at least one ) wafe -included in the table.. 

However,' thes'e tables proved to be too large to conveniently read so a 

routine was"- a'dded "to delete any bug if there was another bug that 

accounted for the same set of answers as well as spme others. ' . ^ 

' ' '', ' -t , . ' . ■ 

•Analysis of a Bug- Comparisc^n Table j . . - ' V 

When we are determining whether Nor not^^ student has a particular 
bug, y/and are confirming . evidence that the student ' hgs the 

bug while both " I . an.d. . numbers ^in t.he bug row are disconf irming 
evidencae. We'refer to the^ resul ts ' in^ a bug abmparlso^h'* tab?||^ as ' 
"eviderice" because, there may be seve'ral- possible •'explanations for any • 
particular answer to^^one problem. ^ The student, may have ♦made a 
careless error -while fallowing his bugged procedure, therefore leading 
to a, number in that bug row instead of a He ma-y have, an 

uhmodeled combination' of bugs, only one of which manifested itself and 
resulted in an "1." in the row of the other/b^ug. Or he may have- been 
following a totjally. different procedure or ' no' proc edur e at all, thSt- 
Just happened to give him the -same answer as a bug leading to a 
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would not show itself in i problem unless -there is a 0 in the 
problem's top number (or 0 is generated dur;Lng solution Jjy borrowing 
from a colufif-n wit h a 1 ) > , . ' ' 

(17) An additional case arises when the student does not answer , a: 
pr.oblem. Although none' of our example ' fables .will include this case, 
it is marked with a,"#". is also used in a bug row -^to indicate 

that the bu^ couTdn't do the problem. We saw very little evidence of 
students not doing a problem possibly because the students were given 
as much time' as tney wanted to complete the . tests . » . 
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In a bug row.. ^The final decision on whether or not t'he student is 
using a buggy subpt^ocedjare must Ue made by considering •# all • of the 

evidence from^ the^ testT But how should cpqfllctihg evXi ^ X lQ^ be 

■ '. . . . ^. . ... • • ^ 

weighted and summed? - . . v ^ 

' ' • ' ' ' ' ' ^ ■ " ■• 

Lfet. us consider an analysis of the bug comparison table for" a 

^ . ■, » 

' particular ^r^t^udent, ;tablp 1. '.Both ^ *FORGET/B'ORROW/0VER/BLANKS and. 
•STOPS/BORROW/AT/^ZERO produce the same answer in problems Vl ( 1039-44) 
and 12 (^05-9) the errant student answers. But , Neither has 

particularly good agreement across the rest of the table. Which, if 
either^ misconception was the student - operating under? Our 
inclination is t^ believe that neither bug satisfactorily explains the 
behavior, but /how does one decide in general? To answer these 
questions, we ai^lal yz ed , by . hand , several hundred student's tables like. 
%he one , in Tabl/e 1 .f, ""During these f orma t iy e anal y se s , the tables were 
examined to ferret out students whose behavior was not captured, by afiy 
existing bugs./ The work of these studen'tjs was closely scrutinized for- 
any underlying computational pattern. 'If a pattern could be 
discerned, tlye incprrect subprocedure was defined and this new^ ."bug^:^ 
was added, to the network. During. this formulation period our list of 
bugs grew fr/om J^S^^o 60. ' . 
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Multiple Bu/gs and Their (Nonobvldua) Inte ractions • '■ 

Most /of the 60 subtraction bugs^ discovered during this period 
were ^ priymitive, in, the sense . t^t each • ^r^edef ined . only one 
> sufepracedur e -^In* the: subtraction ri^woi^^V Was it pos^ibieVthat some of 
^the students' berf€vior was due ' to miil^t^ple bugs t'hat we had failed, to 
notice? / To explore this possibility, we -programmed the BUGGY^ 
diagnost/ic system to try all balrs " of ' bugs. That is, buggy 
definitions of two subprocedures were systematically inserted and then 
execut ed .( 1 8 ) This process* turned up • 270 , bug combinations; who>^e 
symptoins were different from any of Vhe primitive bugs and fjOjQjn e^h 
other/in one test of 15 problems. ^ 

(18) /During this process, bugs which were alternati.ve . definitions of 
the / same conceptual pr oc edur e / of course, could/ not be paired. There 
were/ al so cases where one bug would pr eel ud^ another. For example, 
SKAL/LER/FROM/LARGER precludes an;;^ of t^e bu«gs^' in the Ijorrowing 
procedures as borrowing is. never r equired In these cases, a bug can 
prevent jother portions of the network from ever being executed arid 
hence ^^switching in" bugs in tihe unused por^tion * would' be useless as 
long as! the higher bug remained in effect. Rather than an extensive 
analyils of the potential interactions of bugs/ we opted -for the 
sim^rer solution of comparing the bugs via. their symptoms over a^ fixed 
s^t of problems. ^ 
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In prder too illiistrJate th-e' diagnostic power of this generpative 

technique, consider the example of the student wht)se wbrk is shown in 

• " ^ . • ■ ■ ' \ . ' ' ' ■• ' 

Ta^ble 1 . From Table 1 , no ^ di.scernible* pafcterrf. is evident . The 

J • ■ \ ,' , « 

stij\dent's work does however * admit to a. beautifully simple 
characterizatian which is the composite of two prim^itive buga. Table 
2 -shows the new bug comparison table that was generated for this 
student from comparisons using multiple bugs. (19) Notice that the 
comparison line which resulted from the ^cambinat ion of t KT^^'^^^wo^bugs is. 
substantially different' from either of the s'in'fele bu^ lines or even 
from a linear combination of the two comparisons. This is due to the 
nonobvious interactions of the two bufes,' particularly where the 
intermediate products of one of the.'bugs enables or disaT)les. the ot'her 
bug. For example, in problem 9, 0103-64), the "0-n = n"%u^ alone will 
not manifest itself because the. borrow/ from the!^first colvimri will have 
changed the 0 in the second ocylumn fo 9. ^ However, . the 
"3tops-bor row-at-zero" bug(20) has the side effect; of not changing the 
0 to 9, .and hqnce enables the "0-n=0" bug. In' general, the 



'interactions between bugs can be arbitrarily complex, 
teacher's diagnpst^ic t^sk very difficult.- 



This can make a 




■' V:? . Table 2 
Multiple Bug Cpmparison Table 



8 
3 



99 
79 



353 633 
3M2 221 



81 
17 



4769 25*7 6523 1 03 73t5 
' ' " 64 6536 



5 20 1 1 412- 
Student arisw&rs: 



0 161 1280 
64 4769 "96 5243 



- 416 



39 
169 



779 
738 



(•6lF'F/0-N = N »STOPS/BORROW/'AT/.Z.ERO) : 
« « «« «««« 

•ADD/INSTEADDF/SUB: 

1 r 1.7IB 695 854 



««« 



839 

7803 167 13851 



lOJg 7,05 10038 1 0060 



4319 



98 



. 995 6.96" 5719 9962 
1095 706 14319 10078 



««« ««« 



««« 



««« 



7001 
94 



6^0 7 
7097 



1083 714 14357 10158 7095' 



JudKlng DlagnogtlQ Credibility. 

Using the multi.ple bug co^m^arison tables gener^ated for about one 

hundred stadents, we Identified which of the students, in our 

judgme^at, wer'e m^^kijag procedural,, as oppose^ to careless or, , random 

erro'rs. During this^ hand-done classificatioiK process , we articulated 

. - ' , -v 

(19) In the. tables, multiple ^b'ugs are shown as the names oT the 

Primitive bugs encl-osed in parentheses. 
20) In the " stops^borrow-at-zero" t)ug, the student doesn't know how 
to .'borrow from a column which has a 0 in the minuend so he doesn't do 
anything to .\it .' He does however ^dd 10 to the column he is, 
processing . ; * ^ * 
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and refined our int*Uit,ive use of" the Evidence from a student's ^entire 

test ; in. order to make -that decisifln. Eventually, our und.er standing 

and description of the process became praecipe enough to be 

comput';eri?ed so that it could be run on all 13pt) students. * 

Q\xr hand-done study ' suggest ed ! six intuitive groupings of 

S'tudent s : . . < ' ' " 

. ^ T. Those students who* got all the problems correct. 

*2. Those students who err^d on "any n^ber of problems but whose 
errors were explained one bug or one bug pair. , " 

3. Those students who clearly exhibited the presence of a; .bugVsbi 
who also exhibited some behavior that was not- explalneid by tTfe 
bug .' . 

4. Those — =^udents who missed only onfe or two (of fifteen 
problems) and in a way not consistent" wiith any b.ug . 

5. Those students who exhibited some, buggy behavior but not 
consistently. . 1, T 

6. Those' students whose /txehavior appeared ryahdom- relative to the 
. known bugs . 

— ■ V' ■ . ■ ' ■-'■'^ "n • V ■ 

Tables 3, 4 and 5 show representative students from„ groups 2, '3, and 

5. The injtuitive justification for these grouping^ gemmed fr-om tjje 

possible tutorial approaches 'a teacher might tak"? to remediate a 

studf^nt. The classes of students and the possible .very general 

tutorl^al approaches we-^saw-are: - ( v 

i. those students who are cbrrQcV or veryV^ nearly correct and 
probably- just, need toore practice if anythflx^ ( Groups 1 and M ) ; 

ii."\hose students ^who are . e3dn4bi ting consistently Incorre.ct 
/behavior and, therefore, wnose remediation may .^profit ably , be 
. yieved. as a procels of "diebugging" t he _ s t ud ent ' s present 
-ali^orithm (Groups 2 and 3l; and • - " <• 

. ;iii. "those students for whom a thorough re-teaching of the . entire 
algorithm appears to he essential ( Groups * 5 «and 6). 



e are -definitely not saying that for all students in Groups 2 and 3 



that the on^ or even the best pedagogy is' to focus oh, the student's 
l>uggy procedure. Instead, we are trying to identify those students 
who are consistently making the same mistake and- for whdm* d.ebugging of^ 
his procedure piay be useful. t^^l*^ 



4' 



39 



- 30 - 



.Example of A' Student Whose Behavior is Well Explained by One Bug." 

The bu«i »BORROW/FROM/ZEirO, ' is that; when borrowing from a column in 
whiph the tjjjp number is 0. the student wri.tes 9 but* does not continue 
borrowing from the next column to the* left.^ . * . . 

8 99 353 633 81 4769 257 6523 „ 1 03 .73.1 5 'l O^,? -.705 -10038 10060 7001 
-3 79 342 221 17 0 161 1280 6^ 65-3.6 ' 9 4319 98 94. 

5 20 1 1 412 64 4769 ' 96 5243 39 77 9 ^95 - -69 6 5719 9962 6907 
Student answers: • _ ; 

- - - - r \ ■ r ■^39/ -,.1995 ■ 7 6V15T19 .109'62 ^7 0 07 

(5 " S O 0 1 ) - '■ ' s • • ., ' ' ^ . ' 

•BORROW/FROM/ZERO-: " . -C . 

• • »' • »,'» « »«« « ««« 796 /»«» •»» »»»• 



Best g.uess: • BORROW/ PROM/ ZERO 
GRQUP=2 . - - 



I 



Table 4 

Example of a Student Who' Exhi bit s a Con'si st eht Bug 
... But, Also Has Other Problems. ' " . 

8"99 353 633'81 4769 257 6523 103 7315 1039* 7"05;,T0038 10060-7001 
3 79 342 221 ,1J 0 161 1280. 64 6536 ., 54 -9 '. 4319 . 98 94 

. 5 20 1 1 4T2 64 47 69 ~96 52 43 ~ 77 9 . 99 5 696 57 19 9962 ■•6907 
Student answers: - ^ 1 ' 
- - - - 7'^ - .-• , - 9 1205* 95 704 10.019 70 6 0'0.7 V-' 

-(5 7 1 0 2) . . ■ ■ ^ 

. (•D1FF/0- =0 •MOVE/OVER/ZERO/BORTiOW) . . 

• • • • » ! . • » ,»»» 809 606 ■»•• •»•• 

• (5 7 -1 0/,2) ^ ^ ' ' ■ " - ' 

( •B'ORROW/ACROSS/SMALLER/ADDIJJG/TEN/EXCEPT/ZERO "DIFF/O-NsO ) : " 

! • ♦.»»» 889 606 ' »4» «««- 

'(2 5 4. 2 2) - • / . ■ . * ' V 

'( ISUB/UNITS/SPECIAL •SMALLER/FROM/LARGER&0-N=0 ) : - 

• • ^* • ! .•'•1 1 6 5363>' V 1015 ! •>« 10042 ! 

(2 4 0 3 6) . " ■ / ■ ■ ' S^- 

( •BORROW/NO/DECREMPMT •QUIT / WHEN / BOXTOM/BbANK ) : 
. •• • ;.9 196 5343 49" 1889 V»» 6 6729 72 "l7 

( 1 5 0 2-7) . ' " - • ' ' - 

•BORROW/NO/DECREMENT: . • - " ■ - 

• • • • ^» 196 5343 '149 1889 1095 706 I6729 1 0072 7 01 7 

( 1 5 0 2- 7 ) • * V - ' . 

•SMALLER/FROM/LARGER: • 

• •) •■■ 76 . .» 116.5363 161 1221 1015 1 432^1 1 003&".-709 3 

Best guess: (•DIF'F/0-N = 0 *M0VE/0VER/2ER0y BORROW) 
GR0,UP = 3 , - 

' , ■ ■ , •;■ • ■ , ■■• ... , \ ■ 
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Table 5 



Example of a Student Who Exhibits Some Buggy Behavior But Not 
Consistently. 



■ ' 1 
8 99 ■ 353 633 81 4769 257, 6523 
3 79 342 221 17 0 l6l 1280 



nil "il 



5 20 11 412 64 4769 
Studen.t answers: 



! I ! 



! 



70 ! 



9.6 5243 39 .779 -'995 
98 !« . 109 ■ 839" 109'5' 



(46203) 
(•DIFF/0-N=N 

-U" 6 2 0 3) 
(•DIFF/0-N=N 



•ST OPS/ BORROW/ AT /ZERO) 
! • ! • 169 



•FORGET/BORROW/OVER/BLANKS): 
! • ! • i39 »,ini 



4 6 10 4) 

•DIFF/0-N = 0&1-N = 1 ".ST OPS/ BORROW/ AT /ZERO) 



7,1 



! 



80 9 



(^1^5,0^1 8) 



ZEJIO/I.NSTEADOF/BORROW : 
• • • • • 1^6 5303 100 1000 1005 

GR0UP = 5 * . ' - 



7 0 5' Too 38 100 60, 7 00 1 
9 -431 98 . 94 

696 57 19^ 9962 

706 14319 700 



10019 100-tt) 



69 07 
001 



MO 7 8 7097~ 



107 8 7 007 



« « « 



70D 1002.0 lOOO'O 7000 



Classification Algorithm • - . 

,The algorithm that we eventually converged on for assigning a 
student to one of the six major categories defined on p. 28 is rather 
Involved, so som'e readers may prefer to skim the following disxJXission,. 
If thr-e student erred on at least' one problem, each bug was rated 
according to how well -it accounted for this behavior, T,hi-s rating 
results in a g,roup number for each bug,(21) ThQ rating of each<" bug 

depends on the number of answers falli^ng into each of five groups: 

' * - . * • ■ 

i, those student answers for which tjie bug predicts the student's 
incorrect answer (the number of "»»»"S' appearing in the. bug.'s 
linereferredtoasN***); 

'li, those st udent . answer s for which the bug predicts the student's 
correct answer ( the' number 3of "»"s ; 

lii. thoge student answers for which the bug predicts the correct 
answer but which the student answered incorrectly (the number 
of "!"s N! ) ; 

iv. those student answers for which the bug predicts an incorrect 
answer but which the student answered correc-tly (Nr); 

V. those student answers for which the bug predi,'bts an incorrect 
ajiswer different from the student's incorrect answer (Nw), 



\ 



721) In the ac'tual implementation, the bugs are ordered by the number 
of symptoms accounted for and are ranked from the most promising bug 
until the group number increases. 
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This analysis gives a symptom vector of five numbers (N»»», N», N!, 
Nr, Nw) , examples of which are givenvin Tables 3,*^ and 5' on the lines 
above the bug names, ' 

From the symptom vector, a ^group number corresponding' to the ' six- 
major categories of studertt behavior given on p, 28 is calculated for" 
each buff using the following procedure :( 22 ) 

A bug indicates Group 2 student behavior if it agrees with all 
of the student*' s answers (N! + Nr + Nw = 0) or if it akrees 
more than 75$ o.f the time on problems in which it predicts ar^ 
wrong answer [N 3 x (N! + Nr + Nw)], ^ ' 

./ J. A bug, * indicates^ Group^ behavior*, if it - explains two or more 

stujdent errors ati4 pfTediGts more correct than incorrect 
answers (N»»» >. JhF?^ !+ Nw ' + N!/2), In this formula, those 
problems in which_the bug did not exhibit a .symptom are 
weighted,, by half the' intuition-is that, on these problems, 
^ the student may be exhibiting other bugs- as well, 

A. bug also rated as indicating Group' 3 behavior if it is a 
p'rimatiTe bug (not mul t ipl e )' that , pre d i ct s more than half of 
the student's errors^ and that priedicts erroneous behavior 
more times than it 'fails to 'do so (N*** > N! + Nw and N»»» > 
Nr ) , ' 3 

-A bug indicates Group 5 behavior if it predicts at least 2 
•Incorrect answers (N*** >.2), ^ 

.Otherwise a bug is rated Group 6. 

the student is. assigned the group number of the lowest rated bug. 
If the lowest rated bug is not^ Group 2 or 3 and the student has missed 
only 1 or 2 problems,, he is put in Group 4-, If the student is put in 
"Gro.up 2 or 3> the bug with the lowest group rating (and which accojjnts 
for the most sympto^n^ in cases of ties) is c,hosen as the most , likely ^ 
hy pt>t het i cal student bug, jExanuples of this result can be seen in the\^ 
Orast idne of Tables 3 and 4.. 

. . \ • .. _ ' - \ 

Df^gnos tic R esults for the Nicar aguan Data Base ^ ^ 

The above classification proced ure . was used to analyze the^set of 

test responses for 1 325 4th, 5th, and 6th grader s ,( 2-3 ) A summary of 

the diagnostic classification by grade ±6 given in Table 6, As can be 

.^een,. nearly 40$ of the students exhibited consistently buggy 

behavior; ThisV, figure agrees with a similar result reported by Cox 

(1974.) ,* The ' similarity across grade level may be due to the fact that 

aiddit-ion and subtraction are not presented ag aln ^^^a^t er the 4th grade, 

(22) As we h3ve said, the classification scheme was based primarily on 
empirical studies'. There are intuitive justifications 

(rationalizations) for each of the decisions; however, in the final 
analysis, this algorithm w.-as used because it cla^^ified students . in 
closeaccordancewithourhand-d.oneanalysis. 

(.231 An implementation "note: The entire data analysis program 
including' tne BUGGY subtract ion model s is written^ in INTERLISP,. The 
analysis of all 1 325 students pigainst the 330 bugs required- on the 
order-of 9a minutes of CPU time on a PDP-KL10, 
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Table 6 >^ ^ 

Totals aJ^»¥.ercent ages of Student Classifications 
Groups 2 and 3 are consisteotly following an incorrect procedure (See 



Grade 


1 ^ 
No.($) 


\ \ 2''. , 


3 ■ 


■ ; 4 


5 


6 ' 


Totals 




10 >^ 


101 

/^O .5 


93 

.9 


31 

/^o.6 


197 ^X 
^/i9 .5 


72 

^/i4 .7 


504 ^^Z' 

/foo .b 


? 




86 


73 


.0 


132 


60 


399 ^^ 
-'<00 .0 


6 


17 


88 
/21 .3 


64 
/15 .6 


47 

.6 


131 


75 
^^8.2 


^"fOO .0^ 


Totals 


37 
X3 .2 


275^<^ 
>^1 .2 


230 

. 8 


116 


460 

.2 


207 

>^ . 1 


/ 



It is interesting to note that the most common bug was "when 
borrowing is required fr^om a column in which the \to'p digit. lis .O, 
chang^e the "0 to a 9 but don't continue borrowing from the next c)>lumn 
to the left"; it occurred 107 times in -the 1325 student tests. 
Appendix 5 gives the frequency of the 20 most c^omm^n bugs. 



Wftat Poes a Test Score Mean? 



/ 



Oae of the "^ramificat ions of this fully '.automatic ■ diagnostic 
^«fechnique^ concerns its ability to score tests based on -what a student 
knows or doesn't know as opposed to scoring it based solely on the 
number of right and wrong answers on his test. Even with the advances 
in criterion referenced testing, it remains true that a test is simply 
scored by what problems a studeht gets right or wrong. Because of the 
embedded nature of most procedural skills, a student /can get many 
.probsjems wrohgf^imply by having one fundamental underlying bu^ in a 
primitive subprocedure that he/ is using to solve different problems on 
the test. In such a situation, the score that a student gets can bear 
little relationship to the misconceptions that he actually - harbor.sj A 
stud'^ent can r^eceive a low score either because , he ' many 

misconceptjlons each one of whic^h is more or lesS;^ top of . the 

procedural network 'of skills he is using, or he may possess a few, or 



EKLC 



- 34 



3 



everi^ust one, misconception that is deep down inside the internal 
worlcings of the procedural network and which is co'nstantJLy being used 
to compute Intermediate results u1§ed by higher-up subprocedures . 

Current techniques for cor r ec t Ing ' t e s t s do not off^r easy and 
reliable methods of separating these two situations. The diagnostic 
mod-ellng technique" di scussed above cah take the answers that a student 
gives on a test and, through its modeling system, show not only which 
qfuestiohs were answered incorrectly but why ' t hev were incorrectly 
answered. It Is interesting to note that 10/7 of t^e 1325 -"s-tuden't s- 
tested . had a bug in their borrow- from- zej:^o subpr oce)lur e and -^taisse?! 6 
out of the 15 problems 'on the test because of this one underlying bug. 
The chaf ac t er 1 zat ion given by BUGGY is a much fairer evaluation than 
scoring these students 60$ correct. . • 

• ; ' ^ ■. • ^ ' 

A Methodological Tool for Judging ttie Diagnostic Quality of a Test 

The, procedural, network apparatus also provides a methodologicaT 
t'ool for judging the quality or diagnostic capabilities of a given 
test. This allows one to talk about ' how wel 1 the test can discover and 
delineate commo^ misconceptions. Given a test, each bug in the 
network can be used to answer all of the problems. The resulting 
"buggy" responsej3 are then used to partition the bugs: two bugs' are 
put , in the same partition 4/^ they produce the same answers to all of 
the test ' problems . ' Those bugs, in the same equivalence class or group 
are undifferentiated by the test. "Diagnosticity" of^^the test can nbw 
be defined in terms dif. the size of the * resultant equivalence clasises. 
Also, any bugs l|p t'H*^J^^ group as the correct ansWers are not .being 
tested Tor at- all*. Taking the test - in Figure 4, this s-^stem 
discovered' that a student can have' either of two bugs 
( BO'RROW/ACROSS/SAME or N-N= 1 /AFTER/BORROW) and still get 100$ 
correct answers. ' 

Such a sy"^tem could' provide professional test designers with a 
formal tool for e s t abl i shing , t he diagnostic quality of a proposed 
test. Howev^ , our belief is* that pr-of esslonal test designers have 
good intuit/ons about . diagnostic tests . This belief was confirme^ by 
runnlngy^wo standardized natipnal .tests through the 330 subtraction- 
bugs. -One of the tests left only one Ujig ^ unexposed^^-TTi' '^^K problems 
while the other left 4 bugs unexposed in 10 prob^Iems. The unex^asfed> 



bugs were rare for they were^not even found in the Nicaraguan data. 
.There is a difference, however, between exposing a -bug and diagnosing 
it. Exposing ai bug amounts to having at least one problem on- the test 
in^ which the bug^is manifested; diagnosing a. bug req-uj-res having test 
problem^l^hi|c'h differentiate between it and every other bug. 

Using the Artificial Intelligence paradi^^m of generat e-gtnd- test , 
it . would „ bp straightforward to use BUGGY as a diagnostic test 
generator. The problem . genei^ator must produce "interestingly" 
differen't problems. This generation can be done > using important 
features of problems such as the ' number of times borrowing d)s 
necessary, or whether or not a zero appears in the top number. Sets- 
of generated- problems can , then be filtered using the pro^cedural 
network t/o identify bugs which' are not diagnosed. FVom the bugs left 
undiagnosed, features can, be retrieved whidh direct the generation of 
alternative problems to be added »to the test. In this way, a highly' 
diagnostic test can be developed. Furt hermore , since, the answers that 
would be generated by using the bugs are known, the test c'ould" be a 
mult iple^ choice test and still maintain its total diagnost»ic property!- 
Similarly, a real-time adaptive testing system could be created bajed* 
on these tools. 



i . Section IV 

FUTURE .RESEARCH- . ' 

Thf9 paper has. presented some of the problems" that must be faced 
in diag-noslng failures in procedural skills, and has described 'some 
ideas abo.ut the formulation and^ imp'lementation of ' d i agnost ic modeling 
technique^?, that address- ' these problems. It has also- presented some 
novel uses of diagnostic models as^. a gaming/instructional device, as a 
deep-structure test grader, and as a tool to- j'Udge 'the diagnostic^ 
quality of a test or a set of problems. The central idea underlying 
this research is the uge^ of a pr j apedur^l .network as a means of 
representing diagnostic 'models, ^^he critical properties of this- 
representation sc)ieme are its abilities to represent dn appropriately 
structured breakdown of a skill into sutiskills, to make explicit the 
contr9l structures underlying a collection of skills, and to make the 
knowledge encoded this way directly executable. Such a representation 
jg^najDl.es .a*, particular subskill to be eai^ily modified and then^imulated 

^br executed so that the ramifications of the modification can! be 
quickly ascertained The structure of the network . becomes important 
not only 'because it allows ef f i c i en t ' mod| f i cat ion^ut . also becaus/ the 
representation of the modification can be used to contain explanatory 
or remedial mat er iai • ( 24 ) In addition, the structure allows certain 
types of control structure failures to be directly rS^presented in the 

'network and hence articulated i,f necessary. 

Relationships of - Diagnostic Models to O ther Kinds of Structural Models. 

We now »turn ta • a** brief ' look at the p*ast and' Jiurrent work on 
structural models of stutlents a^id h^w they relate , to diagnostic models 
teased on* procedural networks.' -Most of the past and current research 
on" this* subject has. been focused on the intuitively appealing notion 
tbat if one has an explicit, well formulated model of the knowle^dge 
base of an expert for a given set of -skills or problem domain, then 
one can model a particular student * s ^rknowledge as ^a simplif ication of 
the rules comprising the expert ',s procedures (Brown and Burton , 1974 ; 
Collins, Warnock and Passafiume, 1975; Burton and Brown, 1976;' Carr 



( 24 ) C^)ntrast this with the « ( admittedly strawman ) technique o7 
randomly switching instruct. ions- in a machine language program which 
carries out a skill'. Even if a student's behavior could be 
duplicated, the resulting "model" would be worthless as an explanatory 
deviceorasanaidt;^o remediation. 
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and Goldstein , 1 977 ) . Recently, Goldstein has expanded this' concept 
in his Computer Coach research and has coined the term "overlay 
model" for capturing how a st.udent's^ manifested knowledge 'Of skills 
; relates to an expert's knowledge base (Goldstein, 1977 ) . , . \ 

' The work reported in .thi's paper differs' in that' the basic 
modeling technique is based on viewing^ a structural model of the 
student not as a simpl i f icat ion of the expert's rules btFt rather as a 
set of sems^nt ica'll V meaningful deviations from an expert's knowledge' 
base. (25) Each « subskill of the expert i s ' expli c i 1 1 y encoded , along* 
with a set of poten'tial misconceptions of that subskill. The task of 
inferring a diagnostic model jthen becomes one of discovering which set 
of variations or deviations best ..explains the surface U^havior of the 
student. This view is in concert withj^ although more structured than, 
the approach taken 'by Self (1974) in which he models the student as a 
set of modified procedures ' taken) from a procedural expert 
problem- sol v^er ; ■ r ' 

Another clogely "related approach to modeling a studeht'^s 
knowledge -base uses a production rule encoding scheme (Young ( 1 9;T7 ) 
an-d Smith and Sleeman ( 19770.) However, procedural networks ^differ, 
both theoretically and computationally from these eff-o^t s In that they 
* are designed . to make ■ explicit the representatTtyrr — of the . 
control-structure knowledge underlying a macro skill so that it can be 
efficief)tly diagnosed and. explicitly tutored. 

In' the remainder of this section, we present Q,ur view of the more 
promising di rect ion s^ f or research relating to diagnostic models. 

Extensions t&' the Gaming Environment 

» » ». 

* , ■ . ■ ■ ". ■ 

In the second section, we described the BUGG'Y game ^ tftiich was 

designed to introduce the nottidn ' of "buggy" behavior and provide 

^practice in diagnosing it. ► A,lthough this activity was initially 

designed for training student t eachers ;in diaghosing and ar t iculat ing 

pr^%Dedural bugs, it has. also been used as an -activity to^et kids to 

introspect on their already known procedurea^s well as^o >^ncountep^ 

(25 ) Because t he s6 deviations are based on botto the student's in.tended 
goals and the underlying telecJ.ogy; of the subskills, we ..have no 
automatic way to generate them ta^ opposed tjo wh'at could be done if ., 
the deviations were based on the ^ surf ace syntax .of the rules). 
However, ongoing work by theu authors as well a^. Goldstein and Miller 
\1976 ), and Rich ^nd Schrobe (-1976 ) is / directed toward helping to 
dvercome this limi.tation . * v^:. ;; 



the concept of bugs and debugging3 strateg|l>es in' an easily grasped 
cont extJ . ' " ^ • ' o . 



y ^ A limitation, of ' the * current gaming environment is th'at most of 
tfcifhat the players learn while using the game they learn or discover on 
their own . themoment, BUGGY doe § . no * ex pi i c i t t u t or ing ; it simply 

prolrides an * environment\t hat challenges their theories and encourages 
them to articulate their t hought s ^. ( 26 ) The rest of the -learning 
experience occurs either through the sociology of-^ team learning or 
from what a person abstracts ^pn his own;. The pext step m realizing 
the educational potential of the' BUGGY game is to implem^ent an 
intelligent tutor which can' recognize and point out weakness (or 
interesting facets) in a student's debugging strategies .( 27 ) Our 
experiences indicate that such a tutor would be very helpful for 
middle school and remedial .students who often get caught ^in 
unproductive rut,s. The tutor 'could also help focus the student's 
attention on' the structure of the arithmetic procedures themselves . 
It's, worth Acting th^t sopie of ^he tools for, constructing ; an 
i'?it elligent tutor for the BUGGY game already exist in/,th^, fpfrm of the 
test validation techniques described in the . previous section. 
Nevertheless^ these techniques do not provide the right kinc| of 
information for explaining to a student why the. problem he just 
generated had little or no diagnostic value^^._^We are currently 
exploring the kincj of reasoning required to ansv^er "why". . 

An intelligent tutor designed specif i^lly (to help teachers will 
profit from a theory of what makes an underlying bug easy or difficult 
to diagnose. Simple conjectures c^cerning the depth of ts^-he bug from 
the surface don't seem to work, but more sophisticated iql^asures' might. 
It's hard to see' how to predict, "the degree of difficulty in diagnosing 
a particular bug without a precise inf ormat i on- pr oce s(\s ing or cognitive 
theory of how people actually formulate conj ec-tur^s* abput the 
underlying bug or cause of ap error. For example^ one such 't^hebry is 
that people 'walk through their own. algorithm, looking .for places wher^e 
a part, of. the incorrect answer is dij^rent from their ownj and then 

(26 ) As a historical footnote , BUGGY was original-ly developed to 
explore the psychological validity o^f ithe procedural network model for 
complex procedural skills. Duriiig\that investigation we realizecj the 
ped?agogical potential o^f jev-en JJidns simple versi.on of ^UGGY as an 
rnstruct ional medium^. / . ^ ^ ^ t. ^ 

(27)-Fbr examples of Xthe types of tutor in&. see Brown and Burton, 
1975/197,7; , Carr and Gold st e in , M 977 ; Goldstein, 1974; 

/ 4 , * I' 
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try to imagine local modificat ions .to their algorithm that . could 
account for the error. Onder this theory^^ one would* expect bugs that 
involve major modification r.bf the procedure,, 'such as changing t*e 
direction in which columns are processed, to be diffic'ult to diagnose. 
Sin/ilarly one would expect difficulty if a student's algorithm differs 
from 'the •^diagnostician' s Given an adequate theory, the^.4if f icul t 
situationsy can be watched for and corrected through appropriate 
tutiorial c.omtoents during di agnost ic training . < - 

'J. Ar^other extension to the BUGGY environment to encourag^e fu^rt'her 
coloration of the ideas of hypothesif formation and ' debuggi^ is the 
^/development of- a Specialized programming language for writing simple 
• arithmetic procedures. Acfually ' having / students' write procedures 
provides immediate focus on debugging strategies a topic usually 

left until the end in most secondary school programming courses. In 
this limited environment, it shouli be possible to construct a*n 
intelligent programming assistant as well as a' computers-based tutorial 
helper that can aid a student"^ when »he gets stuck". Providing 
students with a languag-e In which they can write their own procedures 
also allows use of a game developed in the SOPHIE environment ( Brown , 
Rubinstein, and Burton, .1*976 ), where one student writes, a^ prS^Qdure 
introducing a t^ug^ and another student tries to discover it by 
presenting test problems. 

o 

Extensions to .the- Diagnostic Svstem ^ v - [ . ) 

' ■\ Concerning the use of. procedural /network^^ as a' tool for 
diagnosing r^al students, we reiterate that the capabilities of -the 
preserft system' are sol'iry^di agnost ic : * no tptoring is 'at tempt e|l The 

^^issUe of wh-at tutorial strategy to use, even when it is known exactly 
what a student is doing wrong, is still an-^pen question. 

-One possible strategy is that, the "expert" portion j6f the 
^procedur^al network could be made articulate in the sense of b^ng-able 
t;b explain and justify the subproc edure s it uses. Since tite system 
would know t he sjbudent ' s b,ug , special problems can be chosen and 
special care can be taken while presenting the steps wVere the 
student's bug is. This feature could also be used to allow st(ud^nt3 
to pose tlxeir own problems, to the system and obtain a running account 

•of the. relevant procedures as the " expert" vsolves ' jrtie problem. Va 
useful' notion for . the ^ articulate expert ma'y be to have additional 



explanation or justification of each symbolic* procedure in the network 
expressed- in terms of a "physical" procedure using manipulative tools, 
such as Dienes** block's. In this way.,^the exec^ion of each symbolic 
procedure copild "^c ause its analogous physical procedure whose execution 
could be displayed on a graphics device, thereby letting th^ student 
see the numeric -or abstract computation unfold in conjunction with a 
physical model of. the computation. This approach directly attacks the 
problem of "getting procedures to take on "m^eaning" for a student: the 
acquisiti^on of meaning is, we believe, accomplished by recognizing 
mappings or relations between the new procedures and existing^ 
procedures o.r experiences. 

.^While' we cohsider the articulation of the expert to ' have'^great 

■ > ^ ■» . 

^promise as a corrective tutorial strategy, it is by no means the only 

possiljle^* such . str^ategy. It is possible that with certain bugs (and 

cef't airt stude.n t s ). , a clear descrip t ion of what is -going wrong may be 

suyfficieht- to allow the student "to correct his problem. Or it jnay be 

, ' ^ .. ^ , ^ 

possible to formulate a senies of problems , (possibly in conjunction 

♦ . ^ (r • 

with .physical models) ; which enable the student to discover his own 

' • . ' . ' 

error. Or it may be^best to. abandon ,his algorithm Crather^than tt\^yin^ ' 

to debug' it) and start over with a. different, simpler al'gorlthm to' 

build th^ student's confidence. ' ' * ^ ' ' 

- / ■> ^ , ^ . 

Generaliza^tion to g^^rategic Knowledge . , . , 

Cautior^^ should he -exercised'^ in generalizing the procedural 
network mo^el to other'' procedural skills. ^ In particular, the aspects 
of knowledge' discussed here -are almQst totally algorithmic in nature, 
containing 'little heuristic or strategic knowledge' in- selecting or 
guiding the execution' of' the- primitives (procedures). Many 
mathematical skills ilivolve an interplay ^: between . strategic . and 
algorithmic knowledge. For example, when'adding two fractions, one 
doesn't necessarily compute a common multiple'^by multiplying together 
the two d^nomina'tors . Instjead, an examinat ionjf is made of -the l 
rel-at l/onships between the t^o denominators such as identity*, obv-^us, 
divisibility of -one by the other, relative grimeness and s6 on On 
the ba\sis of such relationships specialized procedures/are for 
adding the given fractions. The rules underlying th6se decisions can 
have their ovn bugs and therefore^^, these rjales must Joe modelled within 



some representation scheme. Although 'procedural network schemes ca'n 



represent -^such ^decision - ruled, we^believe that other schemes, based 

perhaps mora oji annotated production rules (Goldstein & Grimson, 

1977)1 deserve serious consideration.. We are> currently investigating 
^ ' ' ' ' . • • • • " 

/ a hybrijd approach for modelling fractions' (and their bugs) which 

involves -merging annotated production rules with procedural networks. 

■. • ■ ' * f • . 

^ PsvcfholQgd.cal Validity - 

, An important area for ^exploration ^concerns the psychological 
validity of the^skill decomposition "and buggy variant's in the network. 
Th^t is, how well do,, the data structures and procedure cal^s in the 

" network corre-spond to the structures and skills that we expect people 
to * learn? Fr.om the network designer's po<iht ^ of view,< - the 
psychological validity can be improved ar ^^denigrat ed 'by choo^sing >-one 
structural decomposition instead of .another. ' Determining a 
psychologiqally ^"corr-ect " functional br-eakdown of a skill into its 
subskills Is critical to the behavior of gaming' and diagnostic systems 
using it. If the breakdown of Hhe skill, is not correct, common bugs 
m'ay be difficult to model wh^^Ie tlrii^e suggested * b/ the <%iodel ijay be. 
judged by ^ people to be "unrealistic".. For people playing BUGGY baseVi 
on a nonvaliwi networic, the relationship^^ between ^dts bugs and -the 
beh.avior they observe, in people will ^ften be obscure. Measuring the 
"corr'ectneiss" of a pJarticular network is a problematic issue as there 
are no clear tests of validity. However, issues- such as the ease^or 
^•naturalness" of inclusion of newly discovered bugs and the appearance 

of combinations of bugs within a breakdown gan be investigated. ^ 

C • . 

Finally, we .have left open the entire issue of a- semantic theory 
of how procedures are understood ( and learned) by"" a person and why 
bugs arise ^in the fLrst place,.' The. need for a theory of hovT 
procedures are learned correctly or incorrectly is important for at' 
least tw6 reasons. First, an int er est ing t heor e t ical framework, that 

• accounts for .the entire col^lection of empirically arrived at bugs will/ 
undoubtedly provide insight into how to correct the .teaching procedure 
that produceSd' the bugs, in the ffrst place. Second? such a theory 

• would be^ the next step in a semantically bas^d generative theory of 
•student modeling. ' As we have stated* -earlier , bugs have to be 
hand-coded in^b the- network, ^now. '-One cark imagine gener a t i vely^ 
producing bugs by a set of ^^yntactic transformations (additions, 
deletions, transpo^i t ipns , • etc.) ba^Qd on some appropriate 



representation language* While some bugs can be naturally accounted 
for as "syntactic" bugs, other-s, such as inappropriate' analogy" frim 
otjier operations or incorrect generalization from examples, are best 
explained outside of the , representation itself and hence require a 
^"semantic" theory. On^ of the by-products of the diagnosis of the 
student;data mentioned in the previous section -has been a thorough and 
precise catalogue of bugs arising in one particular skill, 
subtraction. .This network can now be used to S4igg,est and evaluate 
theories about the origin' of • bugs . . ^ * y ^ >j • 
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APPENDIX I 



Sample Diagnostic Test 

VfKich o£ the following students are making consistent procedural 
mistakes and what are their mistakes? . ' 



6tudent 1; 

83 330 
+106 . +187 
189 417 

Explanation; 



89 ' 354 

•H3r2 ^69 
111 \ ^ 313^ 



Student 2: 



94 

+115 

' 119 i 

Explanation; 



498 

+215 
611^ 



77 
+26 
91 



48 
+41 
89 



X 



Student 3;' 

341 
+139 



476 

Explanation; 



758 % 
+296 
944 



.437 
+284 
601 



923 
: +481 



student 4 ; 

109 - ■ 
-f452 
501 

Explai nation ; 



98 
+105 

""303^* 



/ 



- 98 
+ 111 
209 



r 



35 
+64 
99 



Studertt 5; 

352 ' \ 
+18 ... 
360 

Explanation; 



784 
+3080 
6364 



1784 
+3080 
. 7364 



8 

+35 
63 



student 6t 



8372^ 

-657 

6725 

^ Explanation ; 



6527 
-23^4 

3233^ 



893 
-195 
608 



63 
-47 

16 r 



Student 7: 



•913 
-76 
777 

Explanation ; 



53^4 
-797 
4497 



477/ 
. -284 
/ 101 



893 
-195 
718 



Student 8 ; w 

394 
-166 
248 

Explanation : 



77 
-.53 
24 



935 
t 7361 
774 



126 
-117 
29 



Strident 9; 



48 
-15 
43 

Explanation ; 



394 
-166 
340 



57 
-23 
"6? 



^39 
-95 
124 



Student 10: 



305 
-108 . 

-^107 ^ 

Explanation ; 



987 
-320 
667 



340 
-56 
290 



^ 9280 
J -609.0 
3090 
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Appendix 2 



Student Respons es to What Was Learned 



\ 

4» 



In an experiment described in section 2, 
teachers were exposed to the BUGGY, game, 
a list of all their responses to the 
think you learned fr6% this., experience?" 



a group of student 
This appendix provides 
question, "What do you 



I see from this system that you learn from your* mistake's. In a 
certain operation there ^re so many mis-takes that you can make. When 
you learn what\ the mistakes *are you learn ' to v do ^ the operation 
correctly A- 



That cr\ildren' 
Ire a r.n s 
tested 



errors can be a way of diagnosing the way the child 
material. Also it raises ,. quest io^ns about the way a child' is 
both standardizfed and informally,^ 



s4: u d e n t 



have not 
logical 



s errors apd/or misunderstanding of' a concept ,miy 
been due to carelessness but rather involved, a complex * anv^. 
thought process, ^ 

I learned that it is necessary to try many »dif f erent types^'of examples 
to be sure that a . child really understands. Different types. of 
difficulties arise with different probl'ems,^ 



Feedback is 
diflficult 
long as I 



and -post 
the complexity 6f^ 



extremeTy 
for me to 
can piroV^ 

*i n t h e.. 
-s t^u d e n t ' 3 



•Trying to be'at the machine c;^n be challenging 

important In trying to determine the error. It's 

describe the error buj: the machine doesn't care as 
.my *point through examples, " ^ 

Although it ' s hard to te^ll from' these/ pre 
middle is learned a jgreat deal ^about 

errprs, I know that young students can get these 'preconceived notions 
about how to do things and it's very hard to find a pattern to their 
err^w*^ but there is and 1 believe that BUGGY convinced me of > [ i t ] • ^ 

That if you study the errors long enough y'ou cari eve n^tuallv^ come up 
with a reasonable splution as-to why the [error] "is occuj'r ing , 

Through Jookioig carefully at children's math errof^s it is sometimes, 
.possible to^discover a pattern to them. This pattern will , tell, you an 
•area or a concept the child does not understand, 

I learned that there coul'd be more to a child ' s > mistakes Other t^an 
carelessness. . Working with children with ' special needs I have 
encountered many such problems, yet ^ n^ver stopped to analyze what 
could be a sysytematic problem for this, I thank you, - 



Children do 
especially 
an answer, 
these "answers 
they arrived 




problems and they 
number of differefit 



ed to 
toNhelp 
an swer , 



be more 
them to 



are , very 
operations 



difficulty to spot 
are used to come to 



aware of 'how 
correct them 



these children reach 
first by knowing how 



Although many arithmetic errors may be careless, there, may alsb be a 
pattern that the kid i^ locked into. If you pick up on a pattern, you 
can test the child to see if he/she conforms to it and work on it from 
t h oi^e , ' ' , » 

The types of analysis necessary to ",^ebug^' stucJent errprs on th^ test 
(.paper/ pencil ) seems more difficult than with the computer. But that 
doesn't ma-ke any sense , The "anal ysi s.^ ought to be the same. Perhaps 
the computer motivated my analytical ability, 

.' ■ . - ■ ' r~ 

the problems , 
with BUGGY, 



looking for a' 



I found that I have looked closer at 
relationship between the set after working 

How to perceive problems, that don't look too .consistent, a little 
easier. ' -How to have a good. time with a computer, (I've only played^ 
tic-tac-to.e at the Science Museum; and have always wanted to do more). 
Machines 'can be temperamental (wh/en. p^estered by a large number of 
students? ) ' < — , 
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48 ^ 



57 



I learned and was exposed to the many different types of problema\ 
children might have. I never realized the many different ways a child; 
could devise his own, system to do a problem. I am now aware of 

froblems that could arise and I'm sure this will help me [in] my 
uture career as a teacher. 

How to more effectively detect "problems" stydents have with place 
value . ^ , ^ 

That yoii can* find causes of a child's problem without the child's* work 
in front of you. In looking for the ''bug"., " up and down aren't the 
only possibilities, also diagonally. I suppose horizontally also. 
How specific the problem might be only works in one situation-. 

»I have learned several new possible errors students may make in 
computation. I have also learned somewhat how to diagnose these 
errors, i.e. what to look for, and how specific errors can be. 

V ^ • ~ • 

I think I learned more a.bout computers and how to use them. Also I 

learned about diagnosing math -difficulties . It makes me aware of 

problems that children have and they "sometimes think logically, not, 

carelessly as sometimes teacher s • think they do. . 

I learned 'that computers are very complicated pieces of machinery. If 
■ one isn't experienced with the mechanisms, then problem's could result. 
That computers can be an ^sset to the classroom is nOt"^ doubt ed . but I 
think many problems fc'a-n result-. They can- add much to a classroom 
^until they start breaking down . ' ^ • // 

, That there are many problems that you- can diagnose aboyt a child// by 
looki'hg at his Homework. ' ^ • > . .^^ . ,'\ 

'if a child has repeated!/ made, [the] samfs mistakes, it" ia mrShffe easily 
identified if the teachen^ has ah opportunity to try. and majte .[thej 
dame mista^kes* This method can be solved at least quicker than... 

Computers are concise. Information can be gathered and stored for 
reference. ^ • 

lurxed in to picking up mal f uilc t ions, in simple addition and subtraction 
which seemed to be realistiq^ problems. ^ 
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* Appendix' 3 

^ List of all responses to the q^uestion: 

What is your reactionv to BUGGY? 

''fill * 

I think it would l>e a fantastic resource for a school with 
money to spen^, * , • ^ • 

Too early to tell. But the potential seems stupendous 
and see it as a pc^werful future tool, ' — 

I like it. 



lot of 



I enjoyed it 



Working v^lth a partner is good for* being forced to explain (defend) 

your theory [as long as partner requires that]. Useful tool for those 

with piretty good number ability. What about those who don't have good 

feeling for numbers? - ' " * 



Goo.dl l ! ( Foraes one to get very specific ans.wer ^^t^o the problem,' You 
can..be slightly wrong and then, Vather moving way off. base in your 
second theory as to the problem, you pinpoint /modify your first 
(assuming it's almost right). Bad, It's too much fun and I wasn't 
having very professional in mu usage (though under different situation 
I kight) , \. - 

I it 



^think this system is fantastic. It's 



wohderful way to expose 



people (who are involved with children) to the problems children will 
probably ^lave. It might be especi all y ^ use f ul with special learning 
needs children, . ' ' 

It's great! When will it be in my "price^'. ran^e? • / 



As for the' game itself., it wquld have been continued for another 3 
U ho^rs. ^ / 



or 



I ''think it's an .excellent device for trying to diagnose some of the 
difficulties found in mathematics. For a teacher the time element 
having the machine diagnosis would be more practical, 

I t ' s a nice toy , 

The Bug is;great. Makes you stop and think, ' 

I enjoyed the BUGGY^ c^per i^enc.e extensively, Solving or determining 
error s was ..nuagh eas^^^^r on the cppa.puter^-- and fun too! ^ 

I enjoyed working ^wi.tH ^ ffUGGY l^uiT^ w^h^n it breaks down it^is very 
frustrating. This might be difficiilt for children to understand that 
problems with computers do arise. Also it may be complicated for 
.younger children to understand how to use it. High school students 
may enjoy it.tliough, 

. rf ' - 

I think BUGGY 'would be a definite "plus" in the classroom but ri£ht 
now I feel tbere are too many "bugs" with BUGGY, Too many times did 
"BUGGY, go crazy, / I find it amazing though thkt' a machine can hel.p -one 
detect problems,^ It sure is a/bet ter way than the present, 

BUGGY mak.es one li^k^at eachi problem carefully and detect exactly what 
a child cannot ^o/or cahnTrt:<<^orf'prehend withaut formal testing, 

■ ^ ' / / • ■ ■ ' . ' . 

As far a,s BUGG*Y/is concerned!, I had, a "very good time- "playi^ng" with. 
BUGGY, It^j^s quicker and\ somehow easier than pencil and paper,' It 
took less concentration and was definitely more -efficient , Can . tkrts 
be used as a strictly* diagnostic, tool? If so, I think th&t BUGGY" is 
great • / . 
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He's a trip! Seriously, he's fine if you can master him in case 
decides to break /down, ' * - j- ' 

I think BUGGY \is a^good idea and would like to hear more about it. 

p . " -'' ' - ■ _ . ' • 
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•ItWff a program that should be further 'researc,hed and ha,s excellent 
potential . ' . 

Great ex.\jier lence in beginning* to play with computers exercised 
problem focussing without frustrating a child with inadequate 
pF-eparat iori . • • ; ^ 

I think that RiJGGY could be used to sharpen a teacher's awareness of 
different di f f icult i es ^wi t h addition and subtraction. It might be fun 
for the kids-* to play such a gam^ tjDgether. ^ 



AppencWrtK 4 

Descriptions and Examples of Subt ract ion Bufl;s 



This ^ appendix presents descriptions an^d examples for the 55 
simple subtraction bugs which are modeled in the current network. 
Many o%u\ these b^pgs were 'discovered while we were-r-ex^plor ing data from 
1300 4thy>5th, and 6th grade Students. For each bug^>re listed: (i) 
the answer 'it generates for the problem 15300-9522 ; (.ii) the buK's 
name;, (iii) an additional Example where it fails;' (iv) 
description. Many of these bugs can be combiUled' to produce 
syjnptoms ... * ^ 



and fts 
different 



: 105^78: »BORROW/FRO;4yONE/rS/TEN , 913-7a=935 - . . 

^.jaThe student writes 10 when s/he borrows from a column with a 1 in the 
"V^p digits ' . • , 

••95'778: »B0RROW/FROM/ONE/IS/NINE , 113-58=145 

1 When borrowing from 'a column which has a 1 on top, the student treats 
* tha 1 as if it were a 10 • • 



.27998 : »ADD/BO R ROW/ DECREMENT . 143-28=135 

When borrowing is necessary, instead of subtra'ct ing 

of the next column, the student adds 1 %o it. 



from 



the top digit 



2 4822 : »ADD/INaTEADOF/SUB , 
»The student adds instead of 



.118-5 = 123 
subtracts . 



10 to the top 
the top digit 



16888: » BORROW/ NO /DECREMENT , /143-28= 12 5. 

When the student needs to borr(iw,.he adds 
"cirrrent column without subtracting 1 from 
, col umn . " jjjfe 

1687 8:^ •BORROW/ONCE/WIT'HOUT/RECURSE , 1250-1 088 = 262 
The student will only borrow once per problem 
ten tQthe top number if it is smaller but 
next c3lumn to, the left. ' 

, f6 77 8 : »BORROWyONLY/jbNCE , 1250-J088^2 62 
The student will only/ borrow once per pr&t)lem. 
ten to the top nUmben if it is smaller but 
nex^t c'olumn to the left 

^5778 : I^Vforget'/bor/ow/over/blanks, 

The; stuaent borrows^ correctly except 
digits that are q^y^r blanks 



digit of the 
df the n'ext 



After that s/he 
will' not borrow one' 



, After that s/he 
will not borrow one 



will 
from 



will 
from 



add 
the 



add 
the 



1.43-88,= 155 
•he doesn ' t 



take. 1 Trom the top 



14822: »ADD/NdCAR 
The student adds 



wi 



.Y/INSTEADO'F/SUB, .253-108 = 351 
Ithout carrying instead of subt r a ct in g 



1 4^22 : » SMALL ER/FRbM/ LARGER., 25 3-1 1 8= 1 45 
The student subtracits the smaller digit in 
digit regardless ^of/ which is on top 



a column from the larger 



14222: »SMALLER/FF(OM/LARGER&LEFT/TEN/OK, 1083-768 = 325 
-^he student subtracts the smaller digit in each column from the 
regardless of whicn is on top. The exception is* that when 10 is 
, left-most columns /of the top number 10 it is- treated as a 
single digit . 



14200 : »SMALtE/ft/FROM/Li\RGER&0-N = 0 
The student siLDtracts the smaller 
digit regardlySss ofl-^which is on top^ 
digit is 0, X 0 is written as the a; 



203-98=205 - * • . 
.git in each column from. the 
►The except ion f i^s that when 
jwer f or that ^olumn , i.e.. 



I-,N = 0 



larger 
in the 



larger 
the top 



14>Q5J_.^MALLER/FROM/LARGER&0.-N = 0&(fi-O = 0 , , 903-418=505 ^" 
The^Tudent takes the smaller digi/ .in eaq/h column from the 
r'egardless o^* wh.ich As on top. ifji^ither digit is a*0, s/he 
the answer for that column.^ i 

10022: »BORROW/DIFF/0-l/=N&SMALL-LA] 
The student doesn * tj) know how to bori 



^ arger 
writes 



0^ in 
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}E=0, 204-119=110 
. if ., the top digit 



in a column is 



0,. thek^ St udent writes, the bottom digit in answer ( P-N=N ). If 

the top digit is smaller Uhan the bottom digit, "then 0 is written in 
the answer . 

10000 :. ^ •ZERO/INSTEADOF/BORROW, 140-28= 120 . . \ ^ 

The student writes a 0 in any column th.at rec^u^res- borrowing . 

8748: »Mi:xyUP/SIX/AND/NINE, 356-239= 123 

The student gets ^ 6 and 9 mixed up when reading the digits 
in the prdblem , misreading 6 for 9, and 9 for 6 . 

7998,: »B0RR6w/FR0M/LARGER, 143-58=105 . 

When- borrowing fihAm d column, the student borrows from the larger digit 

disregarding whe ther/ i t is, the top or the bottom digit. 

6888 : » BORROW/ ACROSS/ SMALLER/ ADDING /TEN /UNTIL/BL'ANK/ BOTTOM, 71 3-388 = 335 
The student borrows from the next column to the left 'that ha^ a' larger 



jto.p digit. Any intervening columns have 10 added to their top digit 
'«888: ^ »BORROW/ACftOSS/SMALLER/ADDING/TB^, 563-388=185 ^ 



The student will only borrow fro-m a coPbmn in which the top di^it is 
larger than the bottom digit. In the columns h6 skips 
(where the top digit is smaller ) he automatically adds 10 to the top 
digit . / - 

6 822: »BORROW/AC>ROSS/SMALljER/ADDI-NG/TEN/E;xCEPT/ZmO , 204-159 = 55 . 
The Situd'ent borrows from trhe next column to the left which has a larger 
top digit. Any intervening columfts have 10 added to^ their top digit. The 
exception is that .when 0 isi^n top he writes the 
bottom' number in the answer; i .e., 0-N = N. . 

5878: - »BORROw9FROMyZER04LEFT/TEN/OK, 803-508=395 \ I 

•The student changes* 0 to 9 without further borrowing unless the 0 is' 
part of a"" 10 in the left part of the top number. 

5878 : ' »BOEROW/FROM/ZERO . 103-45=158 V ^ / 

When borrowing from a column whose top dig^-t is 0, the student writes 9,* 

but does nt>t continue borrowing from the column to the left of the 0 . 

5878: »aORROW/FROM/ALL/ZERO. 203-98=205 ^ | 

When borrowing from Q, the student writes 9, but does not continue 

borrowing from the column to the" left of the 0 . If there are two O's in 

a row in the top number, both are changed to 9's. 

5878 : *>BORROW/F,ROM/ZERO&LEFT/OK . 305-296=109 • 

When borrowing from 0 the student writes 9 but does not borrow from the 

x^k^t column to the , left like s/he should. The exception ;is that when the 

.0 Is in the 2nd column from the left and the top n-umber has ^on«e more 

digit than, the bottom: then he sees both of the left digits as a 
single number and siabtracts one from' it correctly. 

5'822 : »DIFF/0-N = N, 140-21 = 121 ' ' 

Whenever the top digit in a column is 0, the studeiit writes the bottom 
digit in the answer; i.e., 0-N=N. / \ 

5822: »DIFF/0-N=N&LEFT/TEN/OK„ 908-395=693 

Vhen there is a 0 on top/, the student writes the bottom digit in the 
answer. The exception is when the 0 is part of 10 in the left columns "of 
the top number. x 

5822 : »DIFF/0-N = N&N^Q = 0 , 302-192=290 ^ / 

The student gets 0 when subtracting 0^ from any thing and also gets N . 
taken from 0 is N. ^ J /■ 

5800 : »DlPF/0-N = 0' l40-2t = 120 * 

Whenever the top digit 'in -a column is 0,, the student writes* 0 In the ' 
ahswer; i.e. , 0-N = 0. ^ v * / 

5800 : »DIFF/0-N = 0&1-N=1 , - 323-1 38 = 215 ' , ■ / 

When there is.O or 1 on top, the student writes it in the.answer. 

5800: »DIFf/0-N = 0&N-(3 = 0 , 308-293= 105 
• The student wri,tes 0 in 'the an'ewer when either the top or the bottom 

. digit' is 0 . ' ; ^ .:r: . ^ ' 

' -53-^^' ' ' ■ • ■ - 
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5798 : »B>OPROW/FROM/BOTTOM/INS'nEADOF/ZERO, •203-158 = 65 

When borrowing from a column with 0 on top, th^ student borrows from the 
bottom digit instead of the 0 on top. 'In all" other cases the student 
borrows correctly. % — ' . 

5788': »DON»T/DECREMENT/ZERd , 603-1 38 = 475 " 

When borrowing from a column where the top digit Is 0, the student,^ 
rewrites the 0 as 10 by borrowing from the next column to the left 'but 
forgets to change 10 to 9 when s/he adds 10 to the column \yhich 
orl^lnajly needed. the borrow. - ^ 

. ,5788: »DON»T/DECREMENT/ZERO/EXCEPT/LAST, 10905-89-16=1999 

When borrowing from a column in which the topydi^it'is 0, the student 
rewrites the 0 as 10 by bor rowing from t he^ next joolumn to the left but 
forgets to change 10 to 9 >fheh s7he adds 10 to the 'column w.hich 
originally needed the^ borrow. The exception is that when 10 is in the • 

* left-most columns of the top digit s/he decrement s by 1 ^ 
correctl/ when* t^orrowirig ( changing the 10 to a 9 ) . 

57 88 : •bON*»T /DECREMENT/ ZERO /UTTt I L/BOTTOM/BLA'NK , 304-259 = 55 
T^he student forgets to change 10 to. 9 after borrowing from a colunfn 
which had 'a 0 on tbp. The exception is that when 0 is part of tha left- 
most. part of the top number 1 is decremente*^ correctly. ^ 

57^8: •STOPS/pQRROW/AT/ZERO i' 203-178 = 35 

The student borrows from zero incorrectly*^ He doesn^'t subtract 1 from 

the 0, \although he- adds 10 correctly to the tofi\ digit of the current 

column V . " y ■ ' ' ■ ' 

5780 : »DIFF/0-N = N/WHEN/BORROW/FROM/ZERO, 1 00 3-^892 = 201 . 
The student writes n In the answer when subtracting n from 0 if s/he l-i^^ 
would have to borrow from a column that contains a 0 in top." * / 

577 8 : » BORROW/ A CROSS/ SAME, 603-108 = 405 

The student will not borrow from the top digit of any columns where . 
the top and bottom digits are the same number. Instead, s/he will 
borrow fro^ the top digit of the next colum^n. 

5778 : •BORROW/ONLY/Fi^OM/TOP/SMALLER, 295-87= 1 ia ' ^. . 
The student wi;il^try to borrow only from those columns where the U^ft 
•digit is smaller ithan the bottom digit. If he can'*t find one, thea 
borrowing is done^ properly. ^ 

5778: •BORROW/INTO/ONE=TEN , 321-89=221 

When borrowing into a colu^an whose top digit is 1 ,* the ^ student . ge t s 10 
instead of 1 1 . ; : ' . 

5778: »DIFF/N-0=0, 403-208=105 

Whenever the bottom d^git in a column is 0, the student writes 0 in the ^ 
answer ( N-0=0.) r 

577«: »DIFF/N-N=N, 235-134=131 ^ • ; ' ^ 

.Whenever the. top digit in a column is the same as the bottom digit, the 
student writes that digit as the answer for that column, i.e., N-N=N. 

57^8: *N-N=1/AFTER/B0RR0W. 354-159 = 215 

The studen't gets 1 when Subtracting n from n in a column that has been 
^borrowed from. -That is, the student knows that he doesn^t need to borrow 

to subtract n from n, but he^ feel^ he must do som-ething with the 
^borrow, s/he writes it in -the answer. ' ' \^ . . 

«'577'8: . •QUIT/WHEN/BOTTOM/BI;aNK, 178-59= 19 / - ' . 

The student' stops vorking the problem as soon as the bottom number runs 

out . ■ . ■■ . ' (J ^ ' • 

5778 : »STOPS/BORROW/tfT/k^ULTIPLE/ZERO , 1003-358 = 655 

The student doesn^t borrow from two. zeros in a row. -S/he will juist add' . . 
ten to the column that rieeds it without decrementing anything. 

■5688:- »MOVE/OVER/ZERO/BORROW, 304^75=139 

When, the student negds to. borrow from a column whose* top digit is 0, he 
skips that column and borrows from the next one. 

5678.: «ONCE/BORROW/ALWAYS/BORROW, 662-357 = 205 ^ ^ ^ * . 

Once the student needs to borrow from a column, he continues to borrow - 
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Into every column whetheir he needp to or not. " ^ 

5372: •SMALLER/ijlOM/LARGER/WITHABORROW, "143-24=111 ' * 
When faced with. borrowing, the student decrements the next column 
correrfrtly, but instead 6.f adding ten to the tW digit oT th3 Current 
column, s/he simply subtracts the smaller digit from >€lJeM^ger diftit 
even though the smaller digit on top; ^i^^i ^^^^ 

4222: . •BORROW/ACROSS/SMALLER&SMALLBR/FROM/LARGER, ' 543-358-n5 
In each column where the smaller digit is on top, the student subtracts 
the smaller digit from <the larger after first- bo-rrowing one 
(uniiecessarily in this ca^e ) from^ the first column to the left in which 
the top digit is larger. then feels tha^t he doesn't have to do any - 
more borrowing in the intervening columns. . V 

/^an't: •BORROW/FROM/BOTTOM, 196-68= 148 " '^-x" 

When the .student needs to borrow, he borrows from 'the bottom ? 
digit of the iiext column instead of the top digit of the nex-t column. 

Can't: •BORROW/TREAT/ONE/AS/ZERO, 313-158=145 >* ' 
When borrowings from a column that has 1 on top, the stddent 'writes 
9 and cc^nti-nues to borrow. That is s/he treats 1 as if it were 0 ^ 
because s/he doesn't like to make more O's in the top nutnber. 

Can't: •BORROWAUNIT/DIFF , 196-68=110 

When the student 'needs to borrow, he borrows the difference between the 
bottom, digit and the top digit of the current column. 

Can't: •ALWAYS/BORROW/LEFT J 602-1 37 = 375 ' 

The student always subtracti& all borrows from/ the left-most digit in the 
top number . ( ) \ 



Can't: •-BORROW^/^WONT/RECURSEX 108-99 = Can't 

When forced to borrow from a c^umn whose top digit is 0, the student 
gives up and says he cannot dothe problem. 

Can't: •WRiTE/BOTTOM/LEFT/ALWAYS, 805-88=Can't 

When writing down the problem, the student Jeft justifies the bottom 
number . • - - 

Can't: »DOESN • T/ BORROW , 1 4 3- 1 38= Can • t 

The student can't do a problem that requires borrowing. ^ 



. " Appendix 5* 

^ Bug Frgqu^ngy T.a.U.e. 

The, 20 most frequently ""^^bcurring bugs in a grou(p of 1,325 students 

-i.....^^.. - 1 

57 students tised the procedure: *BORROW/FROM/ZERO 

When borrowing f^rom a column whbse top digit is 0, the student 
writes. 9f but does not continue' borrowing from ihe Qoliimn to the \ 
left of the 0 - ' f 

5^" students u'sed the procidur»e: »SMALLEr7fR0M/LARGER 

The student subtracts the smaller *4^L^;i.t----ffi a column from ^the 
larger digit regardless of which one is on top. 



50 stfbd^ts used the jt)rocedure *BpRROW/FROM/ZERb&LEFT/TEN/OK 

'The student changes 0 to 9 without further borrowing unless the 0 
\.is part of. a 10 In the left,^.part of the top-numb'er. 



34 students used the procedure: ( •DI^F/0-xN = N 

•move/over/zero/borrow) ^ 

Whenever the top digit in a column is 0, the student writes the 
bottoiri digit in the- answer: i.e.,^ 0-N = N, 

When the student needs to Sorrow from a column whpse top 

digit ds 0, he skips that column ai?.^ borrows from the next one. 

C--..-..- — ^- 

14 st^udents used the procedure: ( »DIFF/b-N = N' 
•STOf Sy4B0RR0W/AT/ZER0) 

Whenever the top digit in a column is 0, the' student writes the 
bottom digit in the answer; i.e. / 0-N = N, 

Th*e student borrows from zero incorrectly.. He doesn't V . 

subtract 1 from the 0 although he adds 10 corr^ectly to the tfp 
digit of the current column, \ 



12 students used the procedure: (*DIFF/0-N=0 ' , ' 

•MOVE/OVER/ ZERO /BORROW) 

Whenever the top digit l,n a* column is 0, the student writes O in 
the answer: i.e., 0-N=0. ' * 

"When the student needs to borrow from a column whose top 
digit is 0, he sKips- that cdluan and borrows from the next one.. 



.11 students used the procedure : ^( *BORRpW/FROM/ ZERO *DIFF/N-0 = 0 )* 

Wh^ti borrowing from a column whose top digit is 0,,^the strident 
writes 9,. but does ' npt cont inue bor row^ngc f r om the column to the 
left of the 0 . " . 

Whenever the bottom digit in a column is 0, the student 
writes 0 in the answer; i.e.,^N-0=0. 

- — — — - - ( 

lO^students used the procedure: *DIFF/0-N = 0&N:iO = 0 . 

The"^ student writes O^in the answer when either the top or the 
bottom digit is 0 . ' ^-^«***^ 



13^'Students used the ^procedure: *SMALLER/FROM/LARGER&p-N = 0 

The student subtracts the smaller digit in each column from the 

rarger digit regardless of which one is on top. The exception is that 

when the top 'digit is 0. a 0 is written as the answer 
for tfeat column; i.e., o-N = 0* 



\ 



10 stkitfents uaed the ^rdcedure: ( »B,ORioW/FROM/ ZERO »DfiiP'^0-N = N ) 

When borrowing ijfom aT^column -Whose top digit is 0, the student 
writes 9f but. does not continue borrowing Yrom the column to the 
left of the- On • ' , * , 

Whenever the top digit in a column is 0, the stuSent writes 
the bottom digit in the answer; i.e., 0-N=.N, ^ ' ' ^ 

10 students used the procedure : »MOVE/OVER/^RO/BORROW • 4,^, -/ 

When, the student needs to borrow ifrom^ column whose top digit is 
0 he skips that column and borrows from the next Q^e; 

10 'Students u3ed the procedure: •DIFF/N-0 = 0 

Whenever th6 bottom digit in a column is 0, the student writes 0 
in the answej?*; i . e • , .N-Q = 0 • 

----------------- - - ----------------------- ^ 

'10 students used the procedoire : •DIFF/0-N = N 

^ . , . . ■ ^ X ^ • 

Whene.ver the top digit in a coltimn is 0, the student writes- the 
bottom digjit in the answer; i.e., 0-N = N, [ - 

— : -' --^L- 

9, students used the procedure: »DIFF/0-N = N&,LEFT/TEN70K 

When there is a 0 on top, the student writes 'tlie' bottom digit iry^ 
the. answer. The excefption is when the 0 isj^part of 10 in the le/t 
col'umns of the top number.^ f ' ^ 

8 students used the procedure: *BORROW/FROM/ALL/ZERO 

When borrowing from 0 the student writes 9 but does not 
continue b'orrowing from the column to the left of the 0, If 
there are two O's in a" row in the top number, both a-re changed to 
9 ' s , - * . , . 



